Nov 01 00:20:49.312295 ip-10-0-141-75 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 00:20:49.312306 ip-10-0-141-75 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 00:20:49.312315 ip-10-0-141-75 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 00:20:49.312637 ip-10-0-141-75 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 00:20:59.424844 ip-10-0-141-75 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 00:20:59.424865 ip-10-0-141-75 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0d24a77b0ffe42f29e9d39bd1b9d42b3 --
Nov 01 00:22:48.314086 ip-10-0-141-75 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 00:22:48.659339 ip-10-0-141-75 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:22:48.659339 ip-10-0-141-75 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 00:22:48.659339 ip-10-0-141-75 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:22:48.659339 ip-10-0-141-75 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:22:48.662583 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662503    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.662583 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662582    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.662583 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662586    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662590    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662593    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662602    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662605    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662607    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662610    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662613    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662615    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662618    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662621    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662624    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662626    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662628    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662631    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662636    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662639    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662641    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.662674 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662645    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662650    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662653    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662657    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662660    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662663    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662665    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662668    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662670    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662673    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662675    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662678    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662681    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662684    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662687    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662690    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662693    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662696    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662698    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662701    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.663146 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662703    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662706    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662709    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662711    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662714    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662716    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662719    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662721    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662724    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662726    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662729    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662731    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662734    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662751    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662754    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662757    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662759    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662762    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662765    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662767    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.663612 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662770    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662775    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662778    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662780    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662783    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662785    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662788    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662792    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662794    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662798    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662800    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662803    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662806    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662808    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662811    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662814    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662816    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662819    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.664094 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662821    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662824    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662826    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662829    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662831    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662834    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662836    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662839    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662842    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662845    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662848    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662852    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662856    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662859    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662862    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662865    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662868    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662870    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662873    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.664537 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662877    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662879    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662882    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662884    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662887    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662890    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.662892    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663655    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663662    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663666    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663669    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663672    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663674    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663677    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663680    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663682    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663685    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663687    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663690    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663692    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.664992 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663696    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663699    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663702    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663705    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663708    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663710    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663713    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663715    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663718    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663721    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663723    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663726    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663728    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663731    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663733    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663751    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663754    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663757    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663760    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.665472 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663763    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663766    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663768    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663771    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663773    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663777    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663780    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663783    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663785    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663788    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663790    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663795    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663799    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663801    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663804    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663807    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663809    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663812    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663815    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.665958 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663818    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663820    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663823    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663825    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663828    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663830    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663833    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663838    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663841    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663843    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663846    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663849    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663851    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663854    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663857    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663859    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663862    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663864    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663867    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.666662 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663870    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663873    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663876    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663878    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663881    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663885    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663889    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663892    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663895    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663897    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663900    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663903    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663905    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663908    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663910    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663913    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663916    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663918    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663921    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.667373 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663923    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663926    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663928    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663931    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663934    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663936    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663939    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663942    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663944    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663947    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663952    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663956    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663961    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663963    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.663966    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664439    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664444    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664447    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664450    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.667854 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664452    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664456    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664460    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664463    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664466    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664468    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664471    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664473    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664475    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664478    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664481    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664483    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664486    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664489    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664491    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664494    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664496    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664498    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664501    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.668303 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664504    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664507    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664509    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664512    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664515    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664517    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664520    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664522    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664525    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664527    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664529    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664532    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664535    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664537    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664540    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664542    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664545    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664548    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664550    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664553    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.668792 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664555    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664557    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664560    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664562    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664564    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664567    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664569    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664572    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664575    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664577    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664580    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664582    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664585    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664588    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664590    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664594    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664596    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664599    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664601    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.669332 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664604    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664606    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664609    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664611    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664614    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664616    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664618    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664621    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664623    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664626    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664628    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664631    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664633    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664636    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664639    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664641    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664643    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664646    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664648    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664651    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.669786 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664653    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664656    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664658    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664661    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664663    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664668    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664671    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664674    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664677    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664680    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664682    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664685    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664687    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664690    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664692    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664695    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664699    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664702    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664706    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664708    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.670291 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664711    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664714    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664716    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.664718    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665692    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665702    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665709    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665714    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665718    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665721    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665725    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665729    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665732    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665749    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665753    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665756    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665759    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665764    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665767    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665770    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665773    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665776    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665780    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 01 00:22:48.670790 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665783    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665787    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665789    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665793    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665797    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665800    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665803    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665806    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665809    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665812    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665815    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665818    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665821    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665825    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665828    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665831    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665833    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665836    2576 flags.go:64] FLAG: --enable-server="true"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665839    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665844    2576 flags.go:64] FLAG: --event-burst="100"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665847    2576 flags.go:64] FLAG: --event-qps="50"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665850    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665853    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665856    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 01 00:22:48.671328 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665860    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665863    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665867    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665870    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665873    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665876    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665880    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665883    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665886    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665888    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665891    2576 flags.go:64] FLAG: --feature-gates=""
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665895    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665898    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665902    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665905    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665908    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665911    2576 flags.go:64] FLAG: --help="false"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665913    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665916    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665927    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665931    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665934    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665937    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 00:22:48.671935 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665940    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665943    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665946    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665949    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665952    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665954    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.665957    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.666029    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.666035    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.666038    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667125    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667136    2576 flags.go:64] FLAG: --lock-file=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667140    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667145    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667149    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667161    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667196    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667201    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667205    2576 flags.go:64] FLAG: --logging-format="text"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667208    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667382    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667398    2576 flags.go:64] FLAG: --manifest-url=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667403    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667432    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667437    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 00:22:48.672486 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667442    2576 flags.go:64] FLAG: --max-pods="110"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667445    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667448    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667451    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667454    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667458    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667461    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667464    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667474    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667477    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667480    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667483    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667486    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667488    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667491    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667495    2576 flags.go:64] FLAG: --port="10250"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667498    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667501    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e4f437043c7d2eb5"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667504    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667507    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667510    2576 flags.go:64] FLAG: --register-node="true"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667514    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667518    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667523    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667526    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667529    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 01 00:22:48.673103 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667540    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667544    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667548    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667551    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667553    2576 flags.go:64] FLAG: --runonce="false"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667556    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667559    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667562    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667565    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667568    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667571    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667574    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667577    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667580    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667583    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667586    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667589    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667593    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667596    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667599    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667605    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667608    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667611    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667616    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667619    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 00:22:48.673747 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667622    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667625    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667628    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667630    2576 flags.go:64] FLAG: --v="2"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667635    2576 flags.go:64] FLAG: --version="false"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667640    2576 flags.go:64] FLAG: --vmodule=""
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667644    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.667650    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667822    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667827    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667830    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667833    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667837    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667840    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667843    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667845    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667848    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667850    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667853    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667856    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667859    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.674928 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667861    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667864    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667867    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667870    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667873    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667875    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667878    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667881    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667883    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667886    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667889    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667891    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667894    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667897    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667899    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667902    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667904    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667907    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667910    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667913    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.675430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667916    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667918    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667921    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667923    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667926    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667928    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667931    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667934    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667936    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667939    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667941    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667944    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667947    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667949    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667952    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667954    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667957    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667959    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667962    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.675911 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667966    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667971    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667974    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667976    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667979    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667981    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667984    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667986    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667989    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667992    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667994    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.667997    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668000    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668002    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668004    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668007    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668010    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668012    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668015    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668017    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.676386 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668020    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668023    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668025    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668029    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668033    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668035    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668038    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668041    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668044    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668046    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668049    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668052    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668054    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668057    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668059    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668062    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668064    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668067    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668069    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.676933 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668072    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668075    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668077    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668080    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668083    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668085    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668088    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668091    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668093    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668096    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668099    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668101    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668104    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668106    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668109    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.677370 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.668139    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668175    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668180    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668183    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668186    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668191    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668194    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668199    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668202    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668204    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668207    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668210    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668212    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668215    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668217    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668220    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668223    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668226    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.677720 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668229    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668231    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668234    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668237    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668239    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668242    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668244    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668247    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668249    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668252    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668254    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668257    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668259    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668264    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668266    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668269    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668271    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668274    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668277    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.678174 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668282    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668285    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668287    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668290    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668292    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668296    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668299    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668302    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668304    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668307    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668309    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668312    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668315    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668319    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668323    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668329    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668331    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668334    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.678639 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668336    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668339    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668342    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668344    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668347    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668349    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668352    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668355    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668357    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668360    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668362    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668367    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668369    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668372    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668374    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668377    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668380    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668384    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668388    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.679096 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668391    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668393    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668397    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668399    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668402    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668405    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668407    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668410    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668412    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668415    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668417    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668420    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668422    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668425    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668427    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668430    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668433    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668436    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668438    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.679565 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668441    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668443    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668446    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668449    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668451    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668456    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668458    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668463    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668466    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.668468    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.675585    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.675598    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.676286    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.676292    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676351    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676356    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676359    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676363    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.680060 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676366    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676369    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676373    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676375    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676378    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676380    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676383    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676386    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676388    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676391    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676394    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676397    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676399    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676402    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676405    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676408    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676411    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676414    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676416    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676419    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.680508 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676424    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676426    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676429    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676432    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676435    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676438    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676441    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676443    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676446    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676448    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676451    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676453    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676456    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676459    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676461    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676464    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676466    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676469    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.681023 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676471    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676474    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676476    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676478    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676481    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676483    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676486    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676488    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676491    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676494    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676496    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676499    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676503    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676506    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676509    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676512    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676515    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676517    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676520    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676523    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.681465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676525    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676528    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676530    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676532    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676535    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676537    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676540    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676542    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676545    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676547    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676549    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676552    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676554    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676557    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676559    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676562    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676564    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676567    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676569    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.681971 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676571    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676574    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676576    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676579    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676582    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676584    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676587    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676589    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676592    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676594    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676598    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676603    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676605    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676608    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676611    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676614    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676616    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676619    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676621    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.682430 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676623    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676626    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676629    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676631    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676634    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.676661    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676824    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676829    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676835    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676837    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676840    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676843    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676846    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676849    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676851    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.682919 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676854    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676861    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676864    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676867    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676869    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676872    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676874    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676877    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676879    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676882    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676885    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676887    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676890    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676893    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676895    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676898    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676900    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676902    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676905    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.683296 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676909    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676913    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676916    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676918    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676921    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676923    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676926    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676929    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676931    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676934    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676937    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676940    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676944    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676947    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676950    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676953    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676956    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676959    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676961    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676964    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.683845 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676966    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676969    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676971    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676974    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676976    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676979    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676981    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676983    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676986    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676988    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676991    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676993    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676996    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.676998    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677000    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677003    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677006    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677009    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677011    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.684319 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677014    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677016    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677018    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677021    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677023    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677026    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677028    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677033    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677036    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677038    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677040    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677043    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677045    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677048    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677051    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677053    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677055    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677058    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677060    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.684858 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677063    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677065    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677067    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677070    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677072    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677075    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677078    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677080    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677083    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677085    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677088    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677090    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677093    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677095    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677097    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677100    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677102    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.685465 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677105    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.677129    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677160    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677163    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677166    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677170    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677173    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677175    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677180    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677182    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677185    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677187    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677190    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677192    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677195    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:22:48.685899 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677198    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677202    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677206    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677209    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677212    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677215    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677218    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677221    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677223    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677226    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677228    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677231    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677233    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677236    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677239    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677242    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677245    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677248    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677250    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:22:48.686275 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677253    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677255    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677258    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677260    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677263    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677266    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677270    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677272    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677275    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677277    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677280    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677282    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677284    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677287    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677290    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677292    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677294    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677297    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677299    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:22:48.686729 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677302    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677304    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677307    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677309    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677312    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677314    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677316    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677319    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677321    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677324    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677329    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677331    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677333    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677336    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677338    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677341    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677343    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677346    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677350    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:22:48.687216 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677352    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677355    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677357    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677359    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677362    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677366    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677368    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677371    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677374    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677377    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677380    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677382    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677385    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677387    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677390    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677392    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677395    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677397    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677400    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677403    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:22:48.687692 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677405    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677408    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677410    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677413    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677415    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677418    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677420    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677423    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677426    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677429    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677431    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677435    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677438    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677440    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:48.677443    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.677494    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.679810    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.680711    2576 server.go:1014] "Starting client certificate rotation"
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.680805    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 00:22:48.688198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.680839    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 00:22:48.688696 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.683396    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 00:22:48.688696 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.684497    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 00:22:48.695119 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.695102    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 01 00:22:48.700361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.700343    2576 log.go:25] "Validated CRI v1 image API"
Nov 01 00:22:48.701234 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.701202    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 00:22:48.708816 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.708795    2576 fs.go:135] Filesystem UUIDs: map[17bad3fb-1bb5-4088-a3af-e4d0b253f050:/dev/nvme0n1p3 6d62a7b8-eb63-4c82-9f0d-89fb2e00dfa1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 00:22:48.708892 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.708814    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 00:22:48.715201 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.715091    2576 manager.go:217] Machine: {Timestamp:2026-11-01 00:22:48.71281303 +0000 UTC m=+0.296237415 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3038718 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c514f8976bfdc8bfb159f8558f174 SystemUUID:ec2c514f-8976-bfdc-8bfb-159f8558f174 BootID:0d24a77b-0ffe-42f2-9e9d-39bd1b9d42b3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:f1:39:e8:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:f1:39:e8:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:b6:57:25:1e:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 00:22:48.715201 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.715204    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 00:22:48.715308 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.715290    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 00:22:48.717129 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717108    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 00:22:48.717271 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717133    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-75.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 00:22:48.717315 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717274    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 00:22:48.717315 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717282    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 00:22:48.717364 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717320    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 00:22:48.717364 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.717339    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 00:22:48.718449 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.718439    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 00:22:48.722665 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.722653    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 00:22:48.722814 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.722805    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 00:22:48.723585 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723576    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 00:22:48.723615 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723589    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 00:22:48.723615 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723599    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 00:22:48.723615 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723608    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 01 00:22:48.723704 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723616    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 00:22:48.723959 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.723947    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 00:22:48.726433 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.726411    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 00:22:48.733541 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.733526    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 00:22:48.734494 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734480    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 00:22:48.734541 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734512    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 00:22:48.734663 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734654    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 00:22:48.734695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734668    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 00:22:48.734695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734675    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 00:22:48.734695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734680    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 00:22:48.734695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734685    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 00:22:48.734695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.734691    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 00:22:48.735388 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735380    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 00:22:48.735417 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735390    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 00:22:48.735417 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735397    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 00:22:48.735417 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735403    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 00:22:48.735417 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735412    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 00:22:48.735573 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.735421    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 00:22:48.736179 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.736170    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 00:22:48.736179 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.736179    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 00:22:48.739953 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.739938    2576 server.go:1263] "Started kubelet"
Nov 01 00:22:48.740043 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.740012    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 00:22:48.740154 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.740089    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 00:22:48.740198 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.740178    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 00:22:48.741586 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.741566    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 00:22:48.741605 ip-10-0-141-75 systemd[1]: Started Kubernetes Kubelet.
Nov 01 00:22:48.742873 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.742851    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 00:22:48.746555 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.746528    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 00:22:48.749396 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.749381    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 00:22:48.750000 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.749984    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 00:22:48.750149 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.750126    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:22:48.750776 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750758    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 00:22:48.750860 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750779    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 00:22:48.750860 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750760    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 00:22:48.750919 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750870    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 00:22:48.750919 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750878    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 00:22:48.751005 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.750992    2576 factory.go:55] Registering systemd factory
Nov 01 00:22:48.751036 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751009    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 01 00:22:48.751219 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751207    2576 factory.go:153] Registering CRI-O factory
Nov 01 00:22:48.751269 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751222    2576 factory.go:223] Registration of the crio container factory successfully
Nov 01 00:22:48.751269 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751267    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 00:22:48.751324 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751284    2576 factory.go:103] Registering Raw factory
Nov 01 00:22:48.751324 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.751297    2576 manager.go:1199] Started watching for new ooms in manager
Nov 01 00:22:48.752335 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.752321    2576 manager.go:319] Starting recovery of all containers
Nov 01 00:22:48.760537 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.760497    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 00:22:48.762559 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.762532    2576 manager.go:324] Recovery completed
Nov 01 00:22:48.764640 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.764617    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 01 00:22:48.767470 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.767453    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.768957 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.768935    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.769024 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.768971    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.769024 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.768982    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.769510 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.769497    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 01 00:22:48.769510 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.769508    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 00:22:48.769612 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.769525    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 00:22:48.769612 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.769533    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.770258 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.770245    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.770304 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.770272    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.770304 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.770282    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.771669 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.771658    2576 policy_none.go:50] "Start"
Nov 01 00:22:48.771704 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.771674    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 00:22:48.771704 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.771684    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 00:22:48.773680 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.773670    2576 policy_none.go:44] "Start"
Nov 01 00:22:48.820547 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.820504    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.825477    2576 manager.go:341] "Starting Device Plugin manager"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.825501    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.825510    2576 server.go:87] "Starting device plugin registration server"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.825808    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.825821    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.825988    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.826095    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.826102    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.826647    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.826698    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.834545    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.834566    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.834589    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 00:22:48.837020 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.834622    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 00:22:48.883785 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.883755    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-75.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 00:22:48.883785 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.883759    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-75.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 00:22:48.884912 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.883727    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-75.us-west-1.compute.internal.18e3c3fd2b0b910c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-75.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-75.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:22:48.7399099 +0000 UTC m=+0.323334284,LastTimestamp:2026-11-01 00:22:48.7399099 +0000 UTC m=+0.323334284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-75.us-west-1.compute.internal,}"
Nov 01 00:22:48.926189 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.926122    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.927088 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.927073    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.927153 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.927103    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.927153 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.927113    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.927153 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.927145    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:48.935695 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.935675    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"]
Nov 01 00:22:48.935862 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.935728    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.937210 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.937193    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.937305 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.937220    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.937305 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.937231    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.939401 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.939390    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.939532 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.939517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:48.939595 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.939550    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.940120 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940105    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.940199 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940130    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.940199 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940143    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.940273 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940203    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.940273 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940222    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.940273 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.940232    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.942191 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.942172    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:48.942273 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.942202    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:22:48.942927 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.942911    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:22:48.943005 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.942940    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:22:48.943005 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.942951    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:22:48.959495 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.959472    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 00:22:48.959568 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:48.959540    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dpgpw"
Nov 01 00:22:48.974657 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.974635    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:48.977414 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:48.977395    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.045809 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.045784    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:49.052246 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.052219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.052327 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.052255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.052327 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.052280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.143953 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.143895    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:49.144444 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.144422    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:49.145547 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.145530    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:49.145721 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.145698    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.145859 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:49.145725    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-75.us-west-1.compute.internal\": node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:22:49.150374 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.150360    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152544 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152619 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152619 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152721 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152721 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.152721 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.152660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.207793 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.207729    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:22:49.208720 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.208708    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.269490 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.268140    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:22:49.279474 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.279452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.282804 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.282775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:22:49.681249 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.681002    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 00:22:49.681777 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.681333    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 00:22:49.681777 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.681333    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 00:22:49.681777 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.681363    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 00:22:49.681777 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:49.681323    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://abbe3b91050ab48e2ac008ae67e6ea89-9261093490b13aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-75.us-west-1.compute.internal.18e3c3fd2cc73c80\": read tcp 10.0.141.75:55836->3.12.131.48:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-75.us-west-1.compute.internal.18e3c3fd2cc73c80  default   4912 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-75.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-75.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-75.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:22:48 +0000 UTC,LastTimestamp:2026-11-01 00:22:48.927117689 +0000 UTC m=+0.510542074,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-75.us-west-1.compute.internal,}"
Nov 01 00:22:49.724168 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.724135    2576 apiserver.go:52] "Watching apiserver"
Nov 01 00:22:49.750191 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.750169    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 00:22:49.965893 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.965797    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:17:50 +0000 UTC" deadline="2028-08-05 13:04:18.769637604 +0000 UTC"
Nov 01 00:22:49.965893 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.965841    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15444h41m28.803799977s"
Nov 01 00:22:49.993819 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.993792    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 00:22:49.994133 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.994116    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lc7vk","kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-gzc59","openshift-network-diagnostics/network-check-target-ccss8","kube-system/global-pull-secret-syncer-q456m","openshift-dns/node-resolver-5d2nq","openshift-multus/multus-additional-cni-plugins-xcdqw","openshift-multus/multus-qpbhj","openshift-ovn-kubernetes/ovnkube-node-6kbnk","kube-system/konnectivity-agent-hcfh5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g","openshift-image-registry/node-ca-qbp6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-vd2ph"]
Nov 01 00:22:49.996720 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.996699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:49.996801 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:49.996783    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:22:49.998773 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:49.998761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.000902 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.000885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:50.001000 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.000953    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:22:50.002974 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.002959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.003030 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.003012    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:22:50.005063 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.005047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.007195 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.007178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.009313 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.009296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.011506 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.011482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.013803 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.013788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.016272 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.016257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.018498 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.018470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.020871 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.020853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.052697 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.051027    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.052697 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.051167    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.052697 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.051266    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\""
Nov 01 00:22:50.052697 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.052320    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 00:22:50.057321 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.057300    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.057420 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.057307    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.057420 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.057351    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\""
Nov 01 00:22:50.058476 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.058537 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9f92\" (UniqueName: \"kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.058537 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.058537 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.058626 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.058626 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.058626 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.058626 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:50.058758 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.058758 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.058758 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.058758 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7w2j\" (UniqueName: \"kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.058944 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.058986    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059102    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059136    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059112    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznhf\" (UniqueName: \"kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059253    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059267    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.059361 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vx6\" (UniqueName: \"kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059449    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.060078 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkdh\" (UniqueName: \"kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45zt\" (UniqueName: \"kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.059983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.060642 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxl4g\" (UniqueName: \"kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tsc\" (UniqueName: \"kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.060994    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 00:22:50.061245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.061141    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\""
Nov 01 00:22:50.063382 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.063367    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.063382 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.063368    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.063599 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.063467    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 00:22:50.063599 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.063467    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\""
Nov 01 00:22:50.063599 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.063545    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 00:22:50.095840 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.095814    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 00:22:50.095840 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.095830    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 00:22:50.098532 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.098516    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\""
Nov 01 00:22:50.101072 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.101054    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 00:22:50.101162 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.101083    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.107239 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.107222    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.107297 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.107279    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.108903 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.108887    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.109068 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.109015    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 00:22:50.109068 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.109041    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 00:22:50.109068 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.109058    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 00:22:50.111092 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.111070    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\""
Nov 01 00:22:50.111092 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.111088    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 00:22:50.111226 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.111091    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 00:22:50.113186 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.113167    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 00:22:50.113267 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.113198    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\""
Nov 01 00:22:50.113267 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.113244    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\""
Nov 01 00:22:50.113375 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.113311    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 00:22:50.113375 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.113368    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tbx2q"
Nov 01 00:22:50.161071 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161046    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.161071 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.161207 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.161207 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.161207 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.161207 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.161207 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vx6\" (UniqueName: \"kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.161424 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161431    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161466    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.161582    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161673    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162138 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.161778    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.661685401 +0000 UTC m=+2.245109778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkdh\" (UniqueName: \"kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161999    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.161914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45zt\" (UniqueName: \"kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.162969 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162471    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162579    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162594    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162776    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.163799 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162798    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162929    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl4g\" (UniqueName: \"kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.162998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.164765 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163037    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163102    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tsc\" (UniqueName: \"kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f92\" (UniqueName: \"kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.165540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.163456    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.163533    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.663488135 +0000 UTC m=+2.246912521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7w2j\" (UniqueName: \"kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.166229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznhf\" (UniqueName: \"kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163972    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.166676 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.163724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.164824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166179    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.167148 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.166425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.222609 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.222281    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:22:50.222609 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.222310    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:22:50.222609 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.222322    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:50.222609 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:50.222397    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.722376493 +0000 UTC m=+2.305800892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:50.223043 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.222977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45zt\" (UniqueName: \"kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:50.223119 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.223050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkdh\" (UniqueName: \"kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.223593 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.223520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vx6\" (UniqueName: \"kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.224241 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.224177    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tsc\" (UniqueName: \"kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.224482 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.224460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.224923 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.224901    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl4g\" (UniqueName: \"kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.225016 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.224986    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznhf\" (UniqueName: \"kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.225554 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.225536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7w2j\" (UniqueName: \"kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.225875 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.225855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f92\" (UniqueName: \"kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.334802 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.334769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:22:50.352803 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.352756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:22:50.372820 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.372790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:22:50.388922 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.388893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qpbhj"
Nov 01 00:22:50.404286 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.404264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:22:50.408981 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.408957    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256267ec21e36d6165c8dc2e9e70ea9f.slice/crio-a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62 WatchSource:0}: Error finding container a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62: Status 404 returned error can't find the container with id a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62
Nov 01 00:22:50.409221 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.409201    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77f3059_b3bd_4359_95fc_2137bab8feef.slice/crio-344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b WatchSource:0}: Error finding container 344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b: Status 404 returned error can't find the container with id 344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b
Nov 01 00:22:50.410406 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.410386    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966f1b0f_5b6a_4dc9_ac24_1ad5de31f629.slice/crio-29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2 WatchSource:0}: Error finding container 29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2: Status 404 returned error can't find the container with id 29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2
Nov 01 00:22:50.413312 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.413267    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15a76296c7098967185b7bb6ae1aa66.slice/crio-8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc WatchSource:0}: Error finding container 8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc: Status 404 returned error can't find the container with id 8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc
Nov 01 00:22:50.414230 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.414208    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:22:50.415066 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.415046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:22:50.416752 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.416719    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b2d432_059c_4433_89dd_f947cf07a66d.slice/crio-96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5 WatchSource:0}: Error finding container 96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5: Status 404 returned error can't find the container with id 96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5
Nov 01 00:22:50.426393 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.426373    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2affb2ab_ec09_43ea_8699_8654990567f3.slice/crio-dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c WatchSource:0}: Error finding container dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c: Status 404 returned error can't find the container with id dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c
Nov 01 00:22:50.434460 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.434440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:22:50.443393 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.443373    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67902a9_529f_4bbe_831b_1e381afd93d8.slice/crio-26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a WatchSource:0}: Error finding container 26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a: Status 404 returned error can't find the container with id 26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a
Nov 01 00:22:50.446504 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.446488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:22:50.451087 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.451075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:22:50.455377 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.455353    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf83a55a_8226_460e_bc4f_ae16df4b4a33.slice/crio-c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991 WatchSource:0}: Error finding container c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991: Status 404 returned error can't find the container with id c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991
Nov 01 00:22:50.459828 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:50.459802    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59a4c1e_8bec_4377_bea2_c86c047de702.slice/crio-8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa WatchSource:0}: Error finding container 8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa: Status 404 returned error can't find the container with id 8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa
Nov 01 00:22:50.634792 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:50.634702    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:52.233477 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.233448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:52.233576 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.233500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:52.233625 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.233598    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:52.233625 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.233598    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:52.233681 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.233662    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.233641551 +0000 UTC m=+3.251673591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:52.233681 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.233677    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.233670648 +0000 UTC m=+3.251702690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:52.298854 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.298825    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:52.334408 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.334368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:52.334571 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.334555    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:22:52.334628 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.334578    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:22:52.334628 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.334592    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:52.334717 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:52.334657    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.334638121 +0000 UTC m=+3.352670173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:52.404677 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.404619    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbp6l" event={"ID":"af83a55a-8226-460e-bc4f-ae16df4b4a33","Type":"ContainerStarted","Data":"c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991"}
Nov 01 00:22:52.405596 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.405571    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcfh5" event={"ID":"2affb2ab-ec09-43ea-8699-8654990567f3","Type":"ContainerStarted","Data":"dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c"}
Nov 01 00:22:52.406600 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.406582    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5"}
Nov 01 00:22:52.407568 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.407551    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vd2ph" event={"ID":"f59a4c1e-8bec-4377-bea2-c86c047de702","Type":"ContainerStarted","Data":"8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa"}
Nov 01 00:22:52.408581 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.408562    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g" event={"ID":"a67902a9-529f-4bbe-831b-1e381afd93d8","Type":"ContainerStarted","Data":"26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a"}
Nov 01 00:22:52.409539 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.409517    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"e15a76296c7098967185b7bb6ae1aa66","Type":"ContainerStarted","Data":"8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc"}
Nov 01 00:22:52.410489 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.410471    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qpbhj" event={"ID":"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629","Type":"ContainerStarted","Data":"29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2"}
Nov 01 00:22:52.411476 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.411458    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerStarted","Data":"a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62"}
Nov 01 00:22:52.412485 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.412465    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerStarted","Data":"344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b"}
Nov 01 00:22:52.681517 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.681412    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:51 +0000 UTC" deadline="2028-05-08 10:56:36.900040116 +0000 UTC"
Nov 01 00:22:52.681517 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.681462    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13306h33m44.218580776s"
Nov 01 00:22:52.700223 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:52.700197    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:22:52.895896 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:22:52.895855    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a11cec8_d010_49d5_876c_fbd012e9da8d.slice/crio-b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134 WatchSource:0}: Error finding container b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134: Status 404 returned error can't find the container with id b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134
Nov 01 00:22:53.240276 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.240234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:53.240463 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.240311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:53.240530 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.240473    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:53.240643 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.240538    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:55.240519937 +0000 UTC m=+5.258552032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:53.240643 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.240614    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:53.240764 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.240649    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:55.240637363 +0000 UTC m=+5.258669407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:53.340796 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.340755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:53.341533 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.340960    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:22:53.341533 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.340978    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:22:53.341533 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.340990    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:53.341533 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.341047    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:22:55.341029464 +0000 UTC m=+5.359061520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:53.400825 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.400792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:53.400993 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.400913    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:22:53.401034 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.400994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:53.401088 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.401067    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:22:53.401167 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.401153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:53.401262 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:53.401237    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:22:53.428594 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.428556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5d2nq" event={"ID":"1a11cec8-d010-49d5-876c-fbd012e9da8d","Type":"ContainerStarted","Data":"b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134"}
Nov 01 00:22:53.440187 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.440150    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzc59" event={"ID":"41fd5adf-add5-424c-b999-ec0c3b16374b","Type":"ContainerStarted","Data":"9dd3fc603c495b51f94680b1657ff79a33cb05e5aac8c5ada0214c7647aa4521"}
Nov 01 00:22:53.682474 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.682354    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:51 +0000 UTC" deadline="2028-08-02 04:59:57.004439434 +0000 UTC"
Nov 01 00:22:53.682474 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:53.682417    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15364h37m3.322026067s"
Nov 01 00:22:55.259030 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.258950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:55.259030 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.259020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:55.259611 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.259148    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:55.259611 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.259206    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:59.25918798 +0000 UTC m=+9.277220034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:55.259611 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.259277    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:55.259611 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.259319    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:59.259300434 +0000 UTC m=+9.277332493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:55.362456 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.359740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:55.362456 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.359907    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:22:55.362456 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.359926    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:22:55.362456 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.359938    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:55.362456 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.359989    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:22:59.359970921 +0000 UTC m=+9.378002978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:55.400793 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.400762    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:55.400954 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.400890    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:22:55.400954 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.400952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:55.401073 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.401034    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:22:55.401127 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:55.401077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:55.401196 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:55.401161    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:22:56.472496 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:56.472396    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="4792ab451d95d922e484bf9872adf26cd909fc1d46b0743a1fdc0c941ecb12e6" exitCode=0
Nov 01 00:22:56.472496 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:56.472487    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"4792ab451d95d922e484bf9872adf26cd909fc1d46b0743a1fdc0c941ecb12e6"}
Nov 01 00:22:57.401060 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.400590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:57.401243 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:57.401180    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:22:57.401314 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.401279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:57.401386 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:57.401354    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:22:57.401445 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.401413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:57.401543 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:57.401521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:22:57.483077 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.483037    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"e15a76296c7098967185b7bb6ae1aa66","Type":"ContainerStarted","Data":"d45fe9605ecb741a11400ae040abbac1afd7fb0b2f0fcd5cd04a9ce016cc7e2c"}
Nov 01 00:22:57.486834 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.486667    2576 generic.go:356] "Generic (PLEG): container finished" podID="256267ec21e36d6165c8dc2e9e70ea9f" containerID="18d0504c721df0fa961099eec1fc07bf070c8046ab2676dfa18b0fb16e738909" exitCode=0
Nov 01 00:22:57.486834 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.486759    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerDied","Data":"18d0504c721df0fa961099eec1fc07bf070c8046ab2676dfa18b0fb16e738909"}
Nov 01 00:22:57.496641 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.495296    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcfh5" event={"ID":"2affb2ab-ec09-43ea-8699-8654990567f3","Type":"ContainerStarted","Data":"1f4579296475c224ba159856ee3a6be25d22106381b1a8c3f2f827785504fb2d"}
Nov 01 00:22:57.599841 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:57.599774    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal" podStartSLOduration=7.5997567010000004 podStartE2EDuration="7.599756701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:57.599150972 +0000 UTC m=+7.617183036" watchObservedRunningTime="2026-11-01 00:22:57.599756701 +0000 UTC m=+7.617788766"
Nov 01 00:22:58.504778 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:58.504728    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerStarted","Data":"01941b1f9118fa7da1c37f5fdbe8ff624143b31600ad209e4157cac2824c47d5"}
Nov 01 00:22:58.508991 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:58.508959    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbp6l" event={"ID":"af83a55a-8226-460e-bc4f-ae16df4b4a33","Type":"ContainerStarted","Data":"da8153b914d934dbc1219f210235600f3d299830ac418793d1b1ee9a66082ab3"}
Nov 01 00:22:58.621686 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:58.621628    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hcfh5" podStartSLOduration=4.009961794 podStartE2EDuration="8.621604928s" totalImagesPullingTime="4.611643134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:57.82803896 +0000 UTC m=+7.846071025" watchObservedRunningTime="2026-11-01 00:22:58.621604928 +0000 UTC m=+8.639636992"
Nov 01 00:22:58.737153 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:58.737056    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" podStartSLOduration=8.737035843 podStartE2EDuration="8.737035843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:58.622477831 +0000 UTC m=+8.640509893" watchObservedRunningTime="2026-11-01 00:22:58.737035843 +0000 UTC m=+8.755067906"
Nov 01 00:22:59.292940 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.292856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:59.292940 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.292925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:59.293175 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.293066    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:59.293175 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.293126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.293106805 +0000 UTC m=+17.311138859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:22:59.293298 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.293213    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:59.293298 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.293250    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.293236521 +0000 UTC m=+17.311268569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:22:59.393293 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.393247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:59.393477 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.393459    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:22:59.393527 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.393484    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:22:59.393527 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.393497    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:59.393597 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.393553    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.393536621 +0000 UTC m=+17.411568678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:22:59.400659 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.400629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:22:59.400829 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.400741    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:22:59.400829 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.400807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:22:59.400994 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.400852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:22:59.400994 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.400896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:22:59.400994 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:22:59.400948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:22:59.519742 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:22:59.519553    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g" event={"ID":"a67902a9-529f-4bbe-831b-1e381afd93d8","Type":"ContainerStarted","Data":"f779f1107161722457ecc18e8752279dfcfc9e3f98e609263a7216b544643ddb"}
Nov 01 00:23:00.526365 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:00.526287    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qpbhj" event={"ID":"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629","Type":"ContainerStarted","Data":"11ae8dfc14a1c663cd120660e7648e05c42737a864bc79e0fabc33dbca9c15f6"}
Nov 01 00:23:00.606390 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:00.606363    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 00:23:00.650779 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:00.650725    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qbp6l" podStartSLOduration=4.938987514 podStartE2EDuration="10.650706233s" totalImagesPullingTime="5.711718719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:58.737577173 +0000 UTC m=+8.755609236" watchObservedRunningTime="2026-11-01 00:23:00.650706233 +0000 UTC m=+10.668738299"
Nov 01 00:23:01.396549 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.396430    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:23:00.606387389Z","UUID":"e12a6b29-08bb-498c-b259-b4a40266ac34","Handler":null,"Name":"","Endpoint":""}
Nov 01 00:23:01.398332 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.398300    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 00:23:01.398437 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.398335    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 00:23:01.401015 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.400992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:01.401172 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:01.401095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:01.401245 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.401207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:01.401317 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:01.401300    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:01.401371 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.401355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:01.401494 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:01.401471    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:01.531100 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.531064    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g" event={"ID":"a67902a9-529f-4bbe-831b-1e381afd93d8","Type":"ContainerStarted","Data":"ed7f37c6e9752d7892e1d5608ff4d67d8316f7d633700a49f8e89b06db4269ac"}
Nov 01 00:23:01.533384 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.533357    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="0249b96330f22440443e52ed9126c3c59292d1f8ec0d4805cea21dd83c6176a3" exitCode=0
Nov 01 00:23:01.533507 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.533394    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"0249b96330f22440443e52ed9126c3c59292d1f8ec0d4805cea21dd83c6176a3"}
Nov 01 00:23:01.535047 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.535016    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5d2nq" event={"ID":"1a11cec8-d010-49d5-876c-fbd012e9da8d","Type":"ContainerStarted","Data":"fdee0b88f5ab0fb3d0920764459a5aab7707fd43276867dc5ca2d6b5781fda67"}
Nov 01 00:23:01.654485 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.654421    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qpbhj" podStartSLOduration=3.340978469 podStartE2EDuration="11.654403281s" totalImagesPullingTime="8.313424812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:00.651017915 +0000 UTC m=+10.669049978" watchObservedRunningTime="2026-11-01 00:23:01.654403281 +0000 UTC m=+11.672435345"
Nov 01 00:23:01.769644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:01.769587    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5d2nq" podStartSLOduration=3.462494778 podStartE2EDuration="11.769570533s" totalImagesPullingTime="8.307075755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:01.768895576 +0000 UTC m=+11.786927639" watchObservedRunningTime="2026-11-01 00:23:01.769570533 +0000 UTC m=+11.787602595"
Nov 01 00:23:02.349940 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.349860    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:02.350817 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.350791    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:02.539367 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.539331    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vd2ph" event={"ID":"f59a4c1e-8bec-4377-bea2-c86c047de702","Type":"ContainerStarted","Data":"e5fb7cf56b9d67c1053f4d833687864d1a20a25135217bccfbaccd7b14f5dec7"}
Nov 01 00:23:02.542986 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.542915    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g" event={"ID":"a67902a9-529f-4bbe-831b-1e381afd93d8","Type":"ContainerStarted","Data":"5ee8aef358710d9b46acb1672cdbeed1e6223aed7319160dbc667de3a67eed03"}
Nov 01 00:23:02.653191 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.653096    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vd2ph" podStartSLOduration=3.858582773 podStartE2EDuration="12.653082399s" totalImagesPullingTime="8.794499626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:02.652641045 +0000 UTC m=+12.670673109" watchObservedRunningTime="2026-11-01 00:23:02.653082399 +0000 UTC m=+12.671114469"
Nov 01 00:23:02.770291 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:02.770238    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g" podStartSLOduration=3.147777693 podStartE2EDuration="12.770218503s" totalImagesPullingTime="9.62244081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:02.769522514 +0000 UTC m=+12.787554629" watchObservedRunningTime="2026-11-01 00:23:02.770218503 +0000 UTC m=+12.788250567"
Nov 01 00:23:03.400421 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.400174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:03.400572 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:03.400510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:03.400572 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.400547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:03.400678 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:03.400616    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:03.400678 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.400668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:03.400745 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:03.400733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:03.546851 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.546816    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzc59" event={"ID":"41fd5adf-add5-424c-b999-ec0c3b16374b","Type":"ContainerStarted","Data":"9e99dd0855d78b96d184eabec5b2fbfd701dc7ec7217234960de8e1676d65d7a"}
Nov 01 00:23:03.549236 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.549207    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="964cf9431d60833d561588cbcc0dfe196c6da20828c966fbcf272b48f0e4fdac" exitCode=0
Nov 01 00:23:03.549365 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.549288    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"964cf9431d60833d561588cbcc0dfe196c6da20828c966fbcf272b48f0e4fdac"}
Nov 01 00:23:03.664304 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:03.664214    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gzc59" podStartSLOduration=3.161227674 podStartE2EDuration="13.664198754s" totalImagesPullingTime="10.50297108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:03.663680837 +0000 UTC m=+13.681712899" watchObservedRunningTime="2026-11-01 00:23:03.664198754 +0000 UTC m=+13.682230897"
Nov 01 00:23:05.400446 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:05.400418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:05.401368 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:05.400534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:05.401368 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:05.400950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:05.401368 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:05.401060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:05.401368 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:05.401149    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:05.401368 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:05.401248    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:05.556854 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:05.556821    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="9d519cc13ad590f32979f401a162949a82ec4d1eb4b033bf3423d288d9ca10e3" exitCode=0
Nov 01 00:23:05.557001 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:05.556874    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"9d519cc13ad590f32979f401a162949a82ec4d1eb4b033bf3423d288d9ca10e3"}
Nov 01 00:23:07.357535 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.357493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:07.358045 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.357563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:07.358045 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.357660    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 00:23:07.358045 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.357661    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:23:07.358045 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.357726    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:23.357709988 +0000 UTC m=+33.375742030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 00:23:07.358045 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.357741    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret podName:3554c745-2c9d-43a8-a83e-50089377ab99 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:23.357734857 +0000 UTC m=+33.375766897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret") pod "global-pull-secret-syncer-q456m" (UID: "3554c745-2c9d-43a8-a83e-50089377ab99") : object "kube-system"/"original-pull-secret" not registered
Nov 01 00:23:07.400789 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.400754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:07.400956 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.400763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:07.400956 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.400891    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:07.400956 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.400763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:07.401073 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.400970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:07.401073 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.401057    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:07.458054 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:07.458016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:07.458238 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.458190    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 00:23:07.458238 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.458208    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 00:23:07.458238 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.458219    2576 projected.go:196] Error preparing data for projected volume kube-api-access-lf9m9 for pod openshift-network-diagnostics/network-check-target-ccss8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:23:07.458328 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:07.458275    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9 podName:fd823994-2ff6-4e3b-b172-97f93135b0df nodeName:}" failed. No retries permitted until 2026-11-01 00:23:23.458261144 +0000 UTC m=+33.476293201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf9m9" (UniqueName: "kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9") pod "network-check-target-ccss8" (UID: "fd823994-2ff6-4e3b-b172-97f93135b0df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 00:23:08.566176 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.565919    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"6929992c9eaed1c9ca6b04f21c6d3fcb401525dc8d2cfa69cda23b5e641d2803"}
Nov 01 00:23:08.566644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.566188    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"ae0f23d25addba797e458e39e73d73ded9ac9fb07c740a2bf2e99145c0d21373"}
Nov 01 00:23:08.566644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.566243    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"6a51f2113dd5cbfc42c344dae1659490cd17d01c06ae2179c30a582732a282fa"}
Nov 01 00:23:08.566644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.566286    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"1734f8c7541796ccd2a42f5dcdb5cf49862824b9760abe7d5c8f9dbe696d6b0f"}
Nov 01 00:23:08.566644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.566322    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"913024da953537e1d62fa5d0fa5003f4eeeee8e2120500d949d4de91021307d3"}
Nov 01 00:23:08.566644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.566360    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"d6c426e55e7b0def96cf0c4b63eb01d31e2b29a7c60ca300ba734c8f37ca6387"}
Nov 01 00:23:08.568571 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.568543    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="bee8d76163fc68ef4e9f65394b7d601affb4c9069873f576cade2c6b0dfd6342" exitCode=0
Nov 01 00:23:08.568694 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:08.568579    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"bee8d76163fc68ef4e9f65394b7d601affb4c9069873f576cade2c6b0dfd6342"}
Nov 01 00:23:09.400674 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:09.400479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:09.400834 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:09.400504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:09.400834 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:09.400742    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:09.400943 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:09.400846    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:09.400943 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:09.400537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:09.401043 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:09.400964    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:09.573094 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:09.573060    2576 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="014c708f2833255012e893de0f3bf0f29fae1b9d0d351c17ceb44121d7e09975" exitCode=0
Nov 01 00:23:09.573480 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:09.573124    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerDied","Data":"014c708f2833255012e893de0f3bf0f29fae1b9d0d351c17ceb44121d7e09975"}
Nov 01 00:23:10.579351 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:10.579309    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" event={"ID":"d77f3059-b3bd-4359-95fc-2137bab8feef","Type":"ContainerStarted","Data":"ed6ec3f0d08c45571f8a6ab98eeb98d43b6f73daf33f8065ff2a42d17fdde7f3"}
Nov 01 00:23:10.582643 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:10.582617    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"2c562907791ddcf55329e4a70851b2fab66508247978e4c3f159c746c3c3ffff"}
Nov 01 00:23:10.799984 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:10.799933    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xcdqw" podStartSLOduration=8.121978316 podStartE2EDuration="20.799920438s" totalImagesPullingTime="12.677942122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:10.749622574 +0000 UTC m=+20.767654636" watchObservedRunningTime="2026-11-01 00:23:10.799920438 +0000 UTC m=+20.817952501"
Nov 01 00:23:11.400941 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.400910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:11.401107 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.400952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:11.401107 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.401010    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:11.401107 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:11.401026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:11.401107 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:11.401084    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:11.401299 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:11.401168    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:11.414032 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.414009    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:11.414149 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.414117    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 00:23:11.414634 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:11.414614    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:13.401055 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.401015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:13.401981 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.401062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:13.401981 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:13.401165    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:13.401981 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:13.401211    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:13.401981 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.401223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:13.401981 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:13.401296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:13.593035 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.593002    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" event={"ID":"d0b2d432-059c-4433-89dd-f947cf07a66d","Type":"ContainerStarted","Data":"a93b97717ee315ccb96932d3b7367cb75fcfc8c1aeed93d38d38f8de0c1269b0"}
Nov 01 00:23:13.593388 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.593368    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:13.609046 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.609021    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:13.873019 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:13.872970    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" podStartSLOduration=7.7797297 podStartE2EDuration="23.872957673s" totalImagesPullingTime="16.093227973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:13.872672028 +0000 UTC m=+23.890704092" watchObservedRunningTime="2026-11-01 00:23:13.872957673 +0000 UTC m=+23.890989775"
Nov 01 00:23:14.597127 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:14.597099    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 00:23:14.597569 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:14.597551    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:14.613218 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:14.613122    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:15.332804 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.332610    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q456m"]
Nov 01 00:23:15.332988 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.332866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:15.332988 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:15.332951    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:15.336093 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.336067    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccss8"]
Nov 01 00:23:15.336235 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.336168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:15.336277 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:15.336241    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:15.336933 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.336914    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc7vk"]
Nov 01 00:23:15.337050 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.337037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:15.337257 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:15.337171    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:15.599398 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:15.599326    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 00:23:16.014085 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:16.014045    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:16.400943 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:16.400859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:16.401083 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:16.400970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:17.400957 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:17.400922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:17.400957 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:17.400967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:17.401372 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:17.401053    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:17.401372 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:17.401151    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:17.615568 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:17.615485    2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk" podUID="d0b2d432-059c-4433-89dd-f947cf07a66d" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 01 00:23:18.400440 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:18.400410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:18.400629 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:18.400521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:19.400661 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:19.400584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:19.401019 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:19.400584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:19.401019 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:19.400692    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccss8" podUID="fd823994-2ff6-4e3b-b172-97f93135b0df"
Nov 01 00:23:19.401019 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:19.400751    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q456m" podUID="3554c745-2c9d-43a8-a83e-50089377ab99"
Nov 01 00:23:20.401301 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:20.401271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:20.401744 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:20.401379    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc7vk" podUID="f532669a-da34-4f31-bbac-a4ee3813e9a5"
Nov 01 00:23:21.130847 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.130818    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeReady"
Nov 01 00:23:21.131013 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.130937    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 00:23:21.283499 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.283423    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d797j"]
Nov 01 00:23:21.335042 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.335005    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qfkjh"]
Nov 01 00:23:21.335218 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.335121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.347718 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.347695    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d797j"]
Nov 01 00:23:21.347836 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.347731    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfkjh"]
Nov 01 00:23:21.347836 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.347809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.387376 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.387343    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 00:23:21.387376 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.387366    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 00:23:21.387578 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.387446    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\""
Nov 01 00:23:21.400053 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400031    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 00:23:21.400160 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400109    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 00:23:21.400160 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400109    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 00:23:21.400261 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400208    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\""
Nov 01 00:23:21.400261 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:21.400261 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.400250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:21.452771 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.452771 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.453197 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.453197 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rllcf\" (UniqueName: \"kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.453197 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.453197 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.452959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqs6\" (UniqueName: \"kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.488355 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.488329    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 00:23:21.488494 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.488447    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 00:23:21.488494 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.488453    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\""
Nov 01 00:23:21.488494 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.488477    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 00:23:21.488585 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.488536    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\""
Nov 01 00:23:21.554281 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.554281 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.554281 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.554563 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllcf\" (UniqueName: \"kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.554563 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:21.554368    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:21.554563 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:21.554448    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.054426519 +0000 UTC m=+32.072458579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:21.554563 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.554563 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqs6\" (UniqueName: \"kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.554838 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554798    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.554983 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.554958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.558069 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.558050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.615819 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.615783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllcf\" (UniqueName: \"kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:21.615952 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.615930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqs6\" (UniqueName: \"kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:21.691947 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:21.691912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:22.059769 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.059679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:22.059913 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:22.059840    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:22.059913 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:22.059905    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:23.059890532 +0000 UTC m=+33.077922572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:22.131524 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.131489    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfkjh"]
Nov 01 00:23:22.135326 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:22.135290    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56ff68b_cfa4_4bdd_b1ea_ab0d1fd5c373.slice/crio-431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274 WatchSource:0}: Error finding container 431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274: Status 404 returned error can't find the container with id 431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274
Nov 01 00:23:22.400571 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.400490    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:22.452977 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.452946    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 00:23:22.453349 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.453047    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\""
Nov 01 00:23:22.617102 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:22.617068    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfkjh" event={"ID":"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373","Type":"ContainerStarted","Data":"431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274"}
Nov 01 00:23:23.066723 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.066677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:23.066878 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:23.066836    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:23.066939 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:23.066932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:25.06691465 +0000 UTC m=+35.084946691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:23.368970 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.368896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:23.368970 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.368948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:23.369166 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:23.369049    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 00:23:23.369166 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:23.369108    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:55.369095454 +0000 UTC m=+65.387127509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : secret "metrics-daemon-secret" not found
Nov 01 00:23:23.371354 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.371334    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:23.469644 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.469612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:23.472922 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.472898    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:23.533862 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.533831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:23.554614 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.554588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:23.909260 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.909227    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q456m"]
Nov 01 00:23:23.927038 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:23.927005    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3554c745_2c9d_43a8_a83e_50089377ab99.slice/crio-0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca WatchSource:0}: Error finding container 0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca: Status 404 returned error can't find the container with id 0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca
Nov 01 00:23:23.960707 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:23.960671    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccss8"]
Nov 01 00:23:23.964408 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:23.964376    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd823994_2ff6_4e3b_b172_97f93135b0df.slice/crio-0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19 WatchSource:0}: Error finding container 0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19: Status 404 returned error can't find the container with id 0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19
Nov 01 00:23:24.624670 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:24.624642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccss8" event={"ID":"fd823994-2ff6-4e3b-b172-97f93135b0df","Type":"ContainerStarted","Data":"0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19"}
Nov 01 00:23:24.625772 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:24.625754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q456m" event={"ID":"3554c745-2c9d-43a8-a83e-50089377ab99","Type":"ContainerStarted","Data":"0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca"}
Nov 01 00:23:25.080647 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:25.080603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:25.080822 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:25.080765    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:25.080884 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:25.080843    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:29.080822516 +0000 UTC m=+39.098854571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:26.636710 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.636621    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfkjh" event={"ID":"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373","Type":"ContainerStarted","Data":"052dec42c855831ac7650902645eeeb34c39cc2733cceebf51dbbed8dcd6e4f4"}
Nov 01 00:23:26.752908 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.752852    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qfkjh" podStartSLOduration=1.6116369210000001 podStartE2EDuration="5.752836019s" totalImagesPullingTime="4.141199098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:26.75182865 +0000 UTC m=+36.769860714" watchObservedRunningTime="2026-11-01 00:23:26.752836019 +0000 UTC m=+36.770868083"
Nov 01 00:23:26.864373 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.864337    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pd52b"]
Nov 01 00:23:26.888327 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.888244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:26.928498 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.926979    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd52b"]
Nov 01 00:23:26.986611 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.986477    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 00:23:26.986736 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.986623    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 00:23:26.986774 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.986739    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 00:23:26.994941 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.994913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrj9r\" (UniqueName: \"kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:26.995058 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.994977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:26.995058 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.995025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:26.995058 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.995050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:26.995206 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:26.995077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.022832 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.022797    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 00:23:27.022832 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.022807    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\""
Nov 01 00:23:27.096367 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrj9r\" (UniqueName: \"kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096540 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096795 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.096996 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.096978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.097066 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.097052    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 00:23:27.097153 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.097119    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:27.597100674 +0000 UTC m=+37.615132729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : configmap "kube-rbac-proxy" not found
Nov 01 00:23:27.097450 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.097431    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:27.097533 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.097485    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:27.597469794 +0000 UTC m=+37.615501841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:27.117750 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.117716    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:23:27.140193 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.140107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.158566 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.158534    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrj9r\" (UniqueName: \"kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.180466 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.180438    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:23:27.193370 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.193324    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cvjqf\""
Nov 01 00:23:27.193507 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.193378    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 00:23:27.244458 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.244341    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 00:23:27.297628 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.297591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.297785 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.297658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.297785 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.297748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.297901 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.297810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.399457 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.399372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.399457 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.399432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.399457 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.399448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.399457 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.399463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.400363 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.400340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.400652 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.400629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.400810 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.400793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.467116 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.467088    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.600439 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.600396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.600630 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.600455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:27.600630 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.600538    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 00:23:27.600630 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.600609    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:28.600593021 +0000 UTC m=+38.618625062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : configmap "kube-rbac-proxy" not found
Nov 01 00:23:27.600630 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.600614    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:27.600825 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:27.600668    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:28.600649247 +0000 UTC m=+38.618681308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:27.723105 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.723072    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"]
Nov 01 00:23:27.739949 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.739916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:27.765044 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.765008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:27.786104 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.786075    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"]
Nov 01 00:23:27.795582 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.795373    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 00:23:27.795582 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.795527    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2ck8t\""
Nov 01 00:23:27.795582 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.795561    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 00:23:27.802267 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.802238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnd8\" (UniqueName: \"kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8\") pod \"migrator-66df5d496b-54cjs\" (UID: \"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:27.902911 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.902877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnd8\" (UniqueName: \"kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8\") pod \"migrator-66df5d496b-54cjs\" (UID: \"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:27.970102 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:27.970075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnd8\" (UniqueName: \"kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8\") pod \"migrator-66df5d496b-54cjs\" (UID: \"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:28.079398 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.079313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:28.171648 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.171620    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:23:28.174815 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:28.174768    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27aafa36_67af_46cb_b144_46b6c60ffb4b.slice/crio-301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189 WatchSource:0}: Error finding container 301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189: Status 404 returned error can't find the container with id 301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189
Nov 01 00:23:28.242019 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.241982    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mn6ww"]
Nov 01 00:23:28.258582 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.258556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.303536 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.303443    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mn6ww"]
Nov 01 00:23:28.311229 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.311193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 00:23:28.311399 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.311256    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 00:23:28.311678 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.311636    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 00:23:28.312156 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.311906    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-plg4w\""
Nov 01 00:23:28.314203 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.312465    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 00:23:28.406249 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.406219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8rl\" (UniqueName: \"kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.406365 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.406278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.406434 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.406403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.465661 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.465625    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"]
Nov 01 00:23:28.473265 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:28.473239    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813dbf2f_9f9e_4aed_bebe_caf642a1e7f3.slice/crio-1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88 WatchSource:0}: Error finding container 1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88: Status 404 returned error can't find the container with id 1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88
Nov 01 00:23:28.507493 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.507458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8rl\" (UniqueName: \"kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.507612 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.507521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.507612 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.507550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.508258 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.508190    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.510049 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.510028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.568976 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.568944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8rl\" (UniqueName: \"kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.580929 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.580905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:28.608713 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.608676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:28.608847 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.608723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:28.608847 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:28.608797    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 00:23:28.608847 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:28.608837    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:28.608980 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:28.608861    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:30.608844892 +0000 UTC m=+40.626876933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : configmap "kube-rbac-proxy" not found
Nov 01 00:23:28.608980 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:28.608895    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:30.60887751 +0000 UTC m=+40.626909557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:28.644925 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.644892    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h6sdm-nnstj" event={"ID":"27aafa36-67af-46cb-b144-46b6c60ffb4b","Type":"ContainerStarted","Data":"301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189"}
Nov 01 00:23:28.646772 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.646737    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccss8" event={"ID":"fd823994-2ff6-4e3b-b172-97f93135b0df","Type":"ContainerStarted","Data":"8251a42cb37ddc5183b5838b76b13e2775e29d65b0ae4274a74b1514a9cb15b8"}
Nov 01 00:23:28.646892 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.646811    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:28.648046 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.648022    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs" event={"ID":"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3","Type":"ContainerStarted","Data":"1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88"}
Nov 01 00:23:28.761199 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.761114    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ccss8" podStartSLOduration=34.322492482 podStartE2EDuration="38.761095643s" totalImagesPullingTime="4.438603161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:28.760840839 +0000 UTC m=+38.778872903" watchObservedRunningTime="2026-11-01 00:23:28.761095643 +0000 UTC m=+38.779127715"
Nov 01 00:23:28.975064 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:28.975033    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mn6ww"]
Nov 01 00:23:28.978955 ip-10-0-141-75 kubenswrapper[2576]: W1101 00:23:28.978925    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbea131_444d_428f_ac3f_90b26919def3.slice/crio-c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624 WatchSource:0}: Error finding container c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624: Status 404 returned error can't find the container with id c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624
Nov 01 00:23:29.113381 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:29.113349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:29.113553 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:29.113481    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:29.113612 ip-10-0-141-75 kubenswrapper[2576]: E1101 00:23:29.113565    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:37.113548917 +0000 UTC m=+47.131580970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:29.653226 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:29.653193    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww" event={"ID":"6dbea131-444d-428f-ac3f-90b26919def3","Type":"ContainerStarted","Data":"c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624"}
Nov 01 00:23:30.243020 ip-10-0-141-75 kubenswrapper[2576]: I1101 00:23:30.242948    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 00:23:30.243159 ip-10-0-141-75 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 00:23:30.264557 ip-10-0-141-75 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 00:23:30.264751 ip-10-0-141-75 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 00:23:30.264809 ip-10-0-141-75 systemd[1]: kubelet.service: Consumed 2.388s CPU time.
Nov 01 00:23:30.274839 ip-10-0-141-75 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 00:23:30.396643 ip-10-0-141-75 kubenswrapper[4493]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:23:30.396643 ip-10-0-141-75 kubenswrapper[4493]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 00:23:30.396643 ip-10-0-141-75 kubenswrapper[4493]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:23:30.396643 ip-10-0-141-75 kubenswrapper[4493]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 00:23:30.399282 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399165    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.399282 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399281    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.399282 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399287    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.399282 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399291    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399294    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399297    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399299    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399302    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399305    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399315    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399318    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399321    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399324    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399327    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399335    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399338    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399341    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399343    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399346    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399349    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399352    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399354    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.399495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399357    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399361    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399365    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399370    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399374    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399378    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399383    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399387    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399393    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399397    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399401    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399405    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399409    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399414    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399417    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399419    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399424    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399428    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399432    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.400047 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399437    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399441    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399445    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399449    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399454    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399457    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399460    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399462    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399465    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399468    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399470    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399473    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399476    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399479    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399483    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399488    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399492    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399496    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399501    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399504    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.400598 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399507    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399510    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399513    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399515    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399518    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399520    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399524    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399527    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399529    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399532    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399535    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399538    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399542    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399547    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399555    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399563    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399568    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399574    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399579    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.401222 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399584    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399588    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399591    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399594    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399596    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399599    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399602    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399604    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399607    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399610    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399614    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399619    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399624    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399628    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399633    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399637    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399641    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399646    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399650    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.401788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399654    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399659    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399664    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399669    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399675    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399679    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.399685    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400654    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400662    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400665    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400668    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400672    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400677    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400682    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400687    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400691    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400695    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400700    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400705    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.402389 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400710    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400714    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400719    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400723    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400726    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400729    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400732    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400734    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400737    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400740    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400742    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400745    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400747    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400750    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400753    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400756    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400760    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400764    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400767    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.402957 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400770    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400773    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400775    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400778    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400781    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400783    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400786    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400789    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400792    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400795    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400797    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400800    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400802    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400805    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400807    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400810    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400812    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400815    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400817    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400821    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.403628 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400825    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400830    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400834    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400838    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400841    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400844    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400846    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400849    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400852    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400855    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400859    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400863    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400867    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400871    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400876    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400880    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400885    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400890    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400894    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400899    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.404281 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400904    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400908    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400913    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400917    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400920    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400923    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400925    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400928    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400931    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400934    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400936    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400938    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400941    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400944    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400946    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400950    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400952    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400955    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400957    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.404988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400960    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400963    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400965    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400970    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400974    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400978    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400982    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400985    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400988    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400990    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400993    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400995    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.400998    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401002    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401005    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401007    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401670    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401678    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401682    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.405564 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401686    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401691    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401695    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401699    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401704    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401708    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401714    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401719    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401723    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401727    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401732    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401737    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401741    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401745    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401751    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401754    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401756    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401759    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401761    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401764    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.406114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401766    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401769    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401771    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401776    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401780    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401785    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401789    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401799    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401802    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401805    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401807    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401810    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401813    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401815    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401818    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401820    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401823    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401825    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401827    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401830    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.406740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401833    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401836    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401840    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401843    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401847    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401849    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401852    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401857    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401862    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401866    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401871    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401875    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401879    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401883    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401888    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401897    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401902    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401907    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401911    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.407375 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401916    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401925    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401928    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401931    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401933    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401937    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401942    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401948    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401955    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401960    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401962    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401965    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401968    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401971    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401973    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401976    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401979    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401982    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401985    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.407980 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401987    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401990    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401992    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401995    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.401997    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402000    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402002    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402005    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402007    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402014    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402017    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402019    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402022    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402024    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402027    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402036    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402039    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402041    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402043    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402048    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.408602 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402054    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402058    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402062    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402065    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.402068    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402208    4493 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402216    4493 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402229    4493 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402233    4493 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402238    4493 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402242    4493 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402248    4493 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402253    4493 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402258    4493 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402267    4493 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402272    4493 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402278    4493 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402283    4493 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402289    4493 flags.go:64] FLAG: --cgroup-root=""
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402294    4493 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402299    4493 flags.go:64] FLAG: --client-ca-file=""
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402307    4493 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402311    4493 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 00:23:30.409316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402322    4493 flags.go:64] FLAG: --cluster-domain=""
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402326    4493 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402332    4493 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402337    4493 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402342    4493 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402359    4493 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402363    4493 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402366    4493 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402369    4493 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402372    4493 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402375    4493 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402378    4493 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402381    4493 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402384    4493 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402388    4493 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402392    4493 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402395    4493 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402398    4493 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402401    4493 flags.go:64] FLAG: --enable-server="true"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402404    4493 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402412    4493 flags.go:64] FLAG: --event-burst="100"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402416    4493 flags.go:64] FLAG: --event-qps="50"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402418    4493 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402423    4493 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 00:23:30.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402426    4493 flags.go:64] FLAG: --eviction-hard=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402430    4493 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402433    4493 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402436    4493 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402439    4493 flags.go:64] FLAG: --eviction-soft=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402442    4493 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402449    4493 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402453    4493 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402458    4493 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402463    4493 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402467    4493 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402472    4493 flags.go:64] FLAG: --feature-gates=""
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402479    4493 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402482    4493 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402491    4493 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402495    4493 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402498    4493 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402501    4493 flags.go:64] FLAG: --help="false"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402504    4493 flags.go:64] FLAG: --hostname-override="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402507    4493 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402510    4493 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402513    4493 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402517    4493 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 00:23:30.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402521    4493 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402527    4493 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402532    4493 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402537    4493 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402541    4493 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402545    4493 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402551    4493 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402556    4493 flags.go:64] FLAG: --kube-reserved=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402564    4493 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402569    4493 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402574    4493 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402579    4493 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402582    4493 flags.go:64] FLAG: --lock-file=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402585    4493 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402588    4493 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402592    4493 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402598    4493 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402601    4493 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402604    4493 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402607    4493 flags.go:64] FLAG: --logging-format="text"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402610    4493 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402613    4493 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402616    4493 flags.go:64] FLAG: --manifest-url=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402619    4493 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402633    4493 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 00:23:30.412020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402639    4493 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402645    4493 flags.go:64] FLAG: --max-pods="110"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402650    4493 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402655    4493 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402658    4493 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402661    4493 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402664    4493 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402666    4493 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402669    4493 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402677    4493 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402680    4493 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402683    4493 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402686    4493 flags.go:64] FLAG: --pod-cidr=""
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402688    4493 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402691    4493 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402695    4493 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402700    4493 flags.go:64] FLAG: --port="10250"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402705    4493 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402709    4493 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e4f437043c7d2eb5"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402715    4493 flags.go:64] FLAG: --qos-reserved=""
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402720    4493 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402725    4493 flags.go:64] FLAG: --register-node="true"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402732    4493 flags.go:64] FLAG: --register-with-taints=""
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402738    4493 flags.go:64] FLAG: --registry-burst="10"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402743    4493 flags.go:64] FLAG: --registry-qps="5"
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402748    4493 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 00:23:30.412756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402751    4493 flags.go:64] FLAG: --reserved-memory=""
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402755    4493 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402758    4493 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402762    4493 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402765    4493 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402768    4493 flags.go:64] FLAG: --runonce="false"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402777    4493 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402780    4493 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402783    4493 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402786    4493 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402789    4493 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402792    4493 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402795    4493 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402798    4493 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402801    4493 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402804    4493 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402807    4493 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402810    4493 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402813    4493 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402816    4493 flags.go:64] FLAG: --system-cgroups=""
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402820    4493 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402832    4493 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402837    4493 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402842    4493 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402850    4493 flags.go:64] FLAG: --tls-min-version=""
Nov 01 00:23:30.413575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402853    4493 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402856    4493 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402859    4493 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402863    4493 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402866    4493 flags.go:64] FLAG: --v="2"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402870    4493 flags.go:64] FLAG: --version="false"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402880    4493 flags.go:64] FLAG: --vmodule=""
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402884    4493 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.402890    4493 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403122    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403142    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403145    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403147    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403151    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403159    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403162    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403164    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403167    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403170    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403173    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403175    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403178    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.414405 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403181    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403183    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403186    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403188    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403191    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403195    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403201    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403206    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403210    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403214    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403221    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403226    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403229    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403232    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403234    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403237    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403240    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403242    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403245    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403247    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.415103 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403250    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403252    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403255    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403257    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403261    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403273    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403278    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403282    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403286    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403290    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403295    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403300    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403304    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403316    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403320    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403323    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403325    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403328    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403332    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.415743 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403335    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403338    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403340    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403343    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403345    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403348    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403350    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403353    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403355    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403358    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403360    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403363    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403365    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403368    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403370    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403373    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403376    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403378    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403383    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.416546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403395    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403399    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403403    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403407    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403410    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403412    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403415    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403417    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403420    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403423    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403427    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403431    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403438    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403442    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403446    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403451    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403455    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403459    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403462    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403466    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.417217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403470    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403474    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403478    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403483    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403487    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403490    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403492    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403495    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403497    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403500    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403502    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403505    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403508    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403511    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403514    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.403553    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:23:30.425367 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403612    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403619    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403624    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403628    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403633    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403637    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403641    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403646    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403652    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403657    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403662    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403666    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403669    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403673    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403678    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403681    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403684    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403686    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403689    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403692    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.425876 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403694    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403697    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403699    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403701    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403704    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403706    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403709    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403712    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403716    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403722    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403727    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403730    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403734    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403738    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403742    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403744    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403748    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403752    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403757    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.426548 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403761    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403768    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403775    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403778    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403781    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403784    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403786    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403789    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403791    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403794    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403796    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403799    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403801    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403804    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403807    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403811    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403815    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403819    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403824    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.427179 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403828    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403833    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403837    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403841    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403843    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403847    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403849    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403852    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403855    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403857    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403859    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403862    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403864    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403867    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403871    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403875    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403878    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403880    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403884    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.427788 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403887    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403889    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403892    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403894    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403897    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403899    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403902    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403906    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403909    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403912    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403915    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403917    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403920    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403922    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403925    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403927    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403932    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403936    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403941    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403945    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.428485 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403950    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403954    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403958    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403961    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403964    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403966    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403969    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403973    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.403980    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.408953    4493 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.408966    4493 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.408983    4493 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.408988    4493 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409075    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409081    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409085    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409088    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409092    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409097    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409101    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.429108 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409104    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409107    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409109    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409112    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409115    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409119    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409123    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409143    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409148    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409152    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409157    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409171    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409177    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409184    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409190    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409195    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409199    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409202    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409205    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409209    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.435641 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409213    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409217    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409222    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409226    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409231    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409235    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409239    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409243    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409248    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409252    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409256    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409260    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409266    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409273    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409278    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409283    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409287    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409291    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409297    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.436457 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409301    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409306    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409310    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409314    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409318    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409329    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409334    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409339    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409342    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409344    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409347    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409349    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409352    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409355    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409357    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409360    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409363    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409365    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409367    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.437278 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409370    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409373    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409375    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409378    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409382    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409386    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409390    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409393    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409395    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409399    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409403    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409407    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409412    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409416    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409420    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409424    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409428    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409432    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409436    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409446    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.438083 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409449    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409451    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409454    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409457    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409459    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409462    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409464    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409467    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409469    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409472    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409474    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409477    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409482    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409486    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409490    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409492    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409495    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409497    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409499    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.438958 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409502    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.409504    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.409550    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410426    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410441    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410447    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410457    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410461    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410466    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410470    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410475    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410479    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410483    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410511    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410515    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410519    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.439800 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410523    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410528    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410536    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410540    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410544    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410549    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410553    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410558    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410562    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410566    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410571    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410575    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410578    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410582    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410586    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410595    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410599    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410603    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410609    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.440513 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410616    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410620    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410625    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410629    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410633    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410637    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410641    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410645    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410654    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410658    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410662    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410689    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410694    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410698    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410702    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410706    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410710    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410715    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410719    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410723    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.441257 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410727    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410736    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410741    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410745    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410750    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410754    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410758    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410763    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410767    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410771    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410775    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410779    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410783    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410788    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410796    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410800    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410805    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410809    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410813    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.442045 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410817    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410821    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410825    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410830    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410857    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410890    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410894    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410898    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410907    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410911    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410915    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410919    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410923    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410928    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410932    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410936    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410944    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410948    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410952    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410956    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.442775 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410965    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410970    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410974    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410978    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410982    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410987    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410991    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.410996    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411000    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411004    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411008    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411012    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411016    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411026    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411032    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.443521 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.411092    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411268    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411274    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411292    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411297    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411302    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411314    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411318    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411323    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411327    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411331    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411336    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411340    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411344    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411349    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411352    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411357    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411361    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411365    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411369    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 00:23:30.444080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411374    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411377    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411380    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411382    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411385    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411388    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411391    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411393    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411398    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411401    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411404    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411408    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411411    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411416    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411421    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411425    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411442    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411447    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411452    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 00:23:30.444685 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411455    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411459    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411468    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411471    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411473    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411476    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411478    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411481    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411484    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411486    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411489    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411492    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411494    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411497    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411500    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411505    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411508    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411510    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411513    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411660    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 00:23:30.445213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411683    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411688    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411695    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411700    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411704    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411709    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411714    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411718    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411723    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411727    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411732    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411742    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411746    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411750    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411754    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411760    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411765    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411769    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411773    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411777    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 00:23:30.445803 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411782    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411786    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411790    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411794    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411804    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411809    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411814    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411818    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411823    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411830    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411834    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411839    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411853    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411857    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411862    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411866    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411875    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411880    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411884    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411890    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 00:23:30.446390 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411894    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411898    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411902    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411907    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411911    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411915    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411929    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:30.411942    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.412076    4493 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414250    4493 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414322    4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414658    4493 server.go:1014] "Starting client certificate rotation"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414769    4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414810    4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:17:50 +0000 UTC" deadline="2028-07-14 22:57:51.112714162 +0000 UTC"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.414869    4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14926h34m20.697848358s"
Nov 01 00:23:30.446984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.415868    4493 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 00:23:30.447504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.417764    4493 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 00:23:30.447504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.419589    4493 log.go:25] "Validated CRI v1 runtime API"
Nov 01 00:23:30.447504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.421124    4493 log.go:25] "Validated CRI v1 image API"
Nov 01 00:23:30.447504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.421968    4493 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 00:23:30.447504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.423672    4493 fs.go:135] Filesystem UUIDs: map[17bad3fb-1bb5-4088-a3af-e4d0b253f050:/dev/nvme0n1p3 6d62a7b8-eb63-4c82-9f0d-89fb2e00dfa1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 00:23:30.447733 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.423695    4493 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc/userdata/shm major:0 minor:65 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/051fe699-0a2b-4aca-83fa-a034cd8788af/volumes/kubernetes.io~projected/kube-api-access-rllcf:{mountpoint:/var/lib/kubelet/pods/051fe699-0a2b-4aca-83fa-a034cd8788af/volumes/kubernetes.io~projected/kube-api-access-rllcf major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a11cec8-d010-49d5-876c-fbd012e9da8d/volumes/kubernetes.io~projected/kube-api-access-b6tsc:{mountpoint:/var/lib/kubelet/pods/1a11cec8-d010-49d5-876c-fbd012e9da8d/volumes/kubernetes.io~projected/kube-api-access-b6tsc major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27aafa36-67af-46cb-b144-46b6c60ffb4b/volumes/kubernetes.io~projected/kube-api-access-26m4r:{mountpoint:/var/lib/kubelet/pods/27aafa36-67af-46cb-b144-46b6c60ffb4b/volumes/kubernetes.io~projected/kube-api-access-26m4r major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2affb2ab-ec09-43ea-8699-8654990567f3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2affb2ab-ec09-43ea-8699-8654990567f3/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3554c745-2c9d-43a8-a83e-50089377ab99/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3554c745-2c9d-43a8-a83e-50089377ab99/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~projected/kube-api-access-gdgpw:{mountpoint:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~projected/kube-api-access-gdgpw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~projected/kube-api-access-pk8rl:{mountpoint:/var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~projected/kube-api-access-pk8rl major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3/volumes/kubernetes.io~projected/kube-api-access-vbnd8:{mountpoint:/var/lib/kubelet/pods/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3/volumes/kubernetes.io~projected/kube-api-access-vbnd8 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/830a85cd-24d7-4745-8274-cf8e043df4d7/volumes/kubernetes.io~projected/kube-api-access-jrj9r:{mountpoint:/var/lib/kubelet/pods/830a85cd-24d7-4745-8274-cf8e043df4d7/volumes/kubernetes.io~projected/kube-api-access-jrj9r major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629/volumes/kubernetes.io~projected/kube-api-access-cxl4g:{mountpoint:/var/lib/kubelet/pods/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629/volumes/kubernetes.io~projected/kube-api-access-cxl4g major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a67902a9-529f-4bbe-831b-1e381afd93d8/volumes/kubernetes.io~projected/kube-api-access-5qkdh:{mountpoint:/var/lib/kubelet/pods/a67902a9-529f-4bbe-831b-1e381afd93d8/volumes/kubernetes.io~projected/kube-api-access-5qkdh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af83a55a-8226-460e-bc4f-ae16df4b4a33/volumes/kubernetes.io~projected/kube-api-access-d7w2j:{mountpoint:/var/lib/kubelet/pods/af83a55a-8226-460e-bc4f-ae16df4b4a33/volumes/kubernetes.io~projected/kube-api-access-d7w2j major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~projected/kube-api-access-r7vx6:{mountpoint:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~projected/kube-api-access-r7vx6 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d77f3059-b3bd-4359-95fc-2137bab8feef/volumes/kubernetes.io~projected/kube-api-access-l9f92:{mountpoint:/var/lib/kubelet/pods/d77f3059-b3bd-4359-95fc-2137bab8feef/volumes/kubernetes.io~projected/kube-api-access-l9f92 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~projected/kube-api-access-2kqs6:{mountpoint:/var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~projected/kube-api-access-2kqs6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f532669a-da34-4f31-bbac-a4ee3813e9a5/volumes/kubernetes.io~projected/kube-api-access-c45zt:{mountpoint:/var/lib/kubelet/pods/f532669a-da34-4f31-bbac-a4ee3813e9a5/volumes/kubernetes.io~projected/kube-api-access-c45zt major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f59a4c1e-8bec-4377-bea2-c86c047de702/volumes/kubernetes.io~projected/kube-api-access-bznhf:{mountpoint:/var/lib/kubelet/pods/f59a4c1e-8bec-4377-bea2-c86c047de702/volumes/kubernetes.io~projected/kube-api-access-bznhf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd823994-2ff6-4e3b-b172-97f93135b0df/volumes/kubernetes.io~projected/kube-api-access-lf9m9:{mountpoint:/var/lib/kubelet/pods/fd823994-2ff6-4e3b-b172-97f93135b0df/volumes/kubernetes.io~projected/kube-api-access-lf9m9 major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/10c6c54287caf879a25607139e9f91866bfb260cf5418cf4e74da0930c77a7db/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2fbd29b6b27af99fe53d80886e8bb4015f0954c6f58da820b4fb1ace25ff7efb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3008a72974990d1ed71dc23a8cf4302a2ec4f6692e72dc6191844b570d4d19d2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/525ffdb8d4823d89ad8d4b86328b4f190a43159f0bf06cdfd6b907baa99bee93/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/bae15bb022cd0650c91962337936e15968bba241e1ef46efd6e788fe9708a637/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/81c8a4feb3d2044e943b6fcaf540067ff410dd29a8a268d1a6a445d50292851d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/25952e4762e0fd208f175263ccfee69919435242438c8f32a6437f13647cfdb5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/55137d61f607ba345604a363e64d7d0f6a1a91243d8e06b3e36f1f19d1be484b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3a92590c3939b0f63eac1e1f95437813bb4ff1af32f79bab968582141ab09378/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/acbe76d5d359138507f41589245496b234d7d019611ce0a3d739e76aeef38946/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/8e50a4804d6fc38b623151e1fe588c963e399c5505c79f6f221ed700ca34d136/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/5c4e6cc7239b895c1c7bdfc85a99d993abf35a9b0a1c246fc5a14ddcba92dc84/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/71a29feaaa73443bd798e61768fd67cc8273cf3f6db96003e354b9f608318d14/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/dfffc94dbb7282d1e0890e51d75d9be11d5f8a07c8df8b52efdab376c3b1a785/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/67bda958a3ec5e63373ed4097e87daf8cbe8563f3a3b89d9a5e87738f21e5296/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/22f8f5d489ee173c6af696ea0d3899876be8680063c3cd9cd3c86b072720b16f/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/342eb0426f168aa34eb9fd83192b5f87e1c82c9ed86a4b6fb16373038cf28d05/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ca57e60bea7ab0d048547c04bba6f67c6cb754cf8ed52624683e7fcca7ea2af8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/587cb7a442cecfc29cc29d21826f182130f745b9f40a819564b2f0f44e703d84/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/59f386ff0d7fdc782fe59d171f55176a3f5e7f2710b69fdc3f351b2b5b982ac1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8eb4344a6de744abb09e0edc466b2fdbd4ef98b390287d0648938b03feabf5e7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/77ebff432ff6097e46621df4864ca73a998549369e3c596c4214e576b2315b09/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/3b354b423fccec730098bc1e51ee84152a3a0e90fd7c03baf022fe1132c444b8/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/1e52fd704cdcb41aaa798410ec58093c1e5ccec5a00b92ab811d2a7ac20aa87a/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/d6230c785d988d431c7be31ca0ecff205d3eb232129254d8a05ff0b92538b25b/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/d523b61c825b6f92eecaa55d22f7eae5097c54263e841818db12246ee8052cbb/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/89a0dcea063cfc21b44ba4766c3b4163c84cc7fee2c1dd7b20e95f511ae36e15/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/7e93c51546475663492bd4bc34f7661c846585b0f918de62aa8904c558751e9e/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/3814ce7739dc50e77dbf15c96c9b3465b99284a5f98125d64bbe1a185b66297c/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/da53dce60a7c89c8930bf6ee2f00f2152fa5804765696d54765a9a02ae1ff82c/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/2665f7299103e1988f35cc5cf8f68360f74f5f4367d309a1f17de82afb61e930/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/7730736c06948b42c4c1120ec8730655720b11c9d1249b0d3dc139b50cf8b55d/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/a468a31dde3cbd6792e6b2d2d84842124e7b61ba0ae1153e06dd5dd6dadc094d/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7d796836ce58ec553d9c18e87ab8acd3314db55495b83f820e35571be29952e1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/136404e67f0d95881972fb7c31eeb1852e86b5597a3e1809680aa14ca25c2507/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7ec466d9c093e0e99d0413198af9f2360e2b1f04919c5a2513a2aec66848cc0f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7ec9b85ee6c2d86373ed82939badfe44d7581a478c30faf6ae81f14f024af105/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/28bfb1fa113786e368d9e669fb6acabbffa23130f36fe4fcbaf1d5a5bfbef48b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/d97b0a39dfda9cddbcda0ab6bd6cd1712f4204d5ca7d86f770a126dd23d78113/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5b1e42d6e05204db3ffb21dff0641725bd33052234d63918b17881355f7c2166/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.430943    4493 manager.go:217] Machine: {Timestamp:2026-11-01 00:23:30.430357239 +0000 UTC m=+0.107944178 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101022 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c514f8976bfdc8bfb159f8558f174 SystemUUID:ec2c514f-8976-bfdc-8bfb-159f8558f174 BootID:0d24a77b-0ffe-42f2-9e9d-39bd1b9d42b3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2affb2ab-ec09-43ea-8699-8654990567f3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf9f1b7b1a8c7bc15e8a4b1df20fd76fea4da0ca94c9241b24cbf57db2bcd7c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f532669a-da34-4f31-bbac-a4ee3813e9a5/volumes/kubernetes.io~projected/kube-api-access-c45zt DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~projected/kube-api-access-gdgpw DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629/volumes/kubernetes.io~projected/kube-api-access-cxl4g DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0815a793cc94d97916d9b1f75ebb1f150ae291720321cd5772619f5c3b5c991/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a67902a9-529f-4bbe-831b-1e381afd93d8/volumes/kubernetes.io~projected/kube-api-access-5qkdh DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/af83a55a-8226-460e-bc4f-ae16df4b4a33/volumes/kubernetes.io~projected/kube-api-access-d7w2j DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27aafa36-67af-46cb-b144-46b6c60ffb4b/volumes/kubernetes.io~projected/kube-api-access-26m4r DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1a11cec8-d010-49d5-876c-fbd012e9da8d/volumes/kubernetes.io~projected/kube-api-access-b6tsc DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26b6a2e3d58aabbdd3c495339c5495925345c41f2a7552d6f5c0f523c426943a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1542a79cb584e32d3cbfe9944f4df4fe7172dc8a72180da48435453d11815d88/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d1fadd2bbb7ea938cf57ee496a0f26f715aa855a4a5e07a480b07cfbdbc7afa/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f59a4c1e-8bec-4377-bea2-c86c047de702/volumes/kubernetes.io~projected/kube-api-access-bznhf DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd823994-2ff6-4e3b-b172-97f93135b0df/volumes/kubernetes.io~projected/kube-api-access-lf9m9 DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9bfdfe29b6796e170847b5b88a0bac00b95fa03f017f368d44f9a7609114624/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/volumes/kubernetes.io~projected/kube-api-access-2kqs6 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0879e31c2f6e6f3e72c10099d17bbf3d4aaa266f9671e4b49034a917d597134/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c05f4b645ae4b2e40e853708c454fe1cdec201e003af2f160eb57f548af8cca/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a5aa953f5d51481bae2d32b898974a55805c82dc123c165fc4db7f66c4f2b19/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3/volumes/kubernetes.io~projected/kube-api-access-vbnd8 DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29923643c721a108f49be4b76b2c159aa666f81420ab5c59447da110752eb3d2/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d77f3059-b3bd-4359-95fc-2137bab8feef/volumes/kubernetes.io~projected/kube-api-access-l9f92 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3554c745-2c9d-43a8-a83e-50089377ab99/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/830a85cd-24d7-4745-8274-cf8e043df4d7/volumes/kubernetes.io~projected/kube-api-access-jrj9r DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6dbea131-444d-428f-ac3f-90b26919def3/volumes/kubernetes.io~projected/kube-api-access-pk8rl DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/41fd5adf-add5-424c-b999-ec0c3b16374b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/344e9f0237fa3a75e8a46ed59b2708130036db113143ddf4f065d8dff348230b/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/431cc09c53a3f910f66b36771e1180b9b6fb49289d890a38b850a02b1e4ae274/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96fce2d48e8090681c4a9b3f941174626659836e1d948e97d9caf91dd2cb87a5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/051fe699-0a2b-4aca-83fa-a034cd8788af/volumes/kubernetes.io~projected/kube-api-access-rllcf DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b2d432-059c-4433-89dd-f947cf07a66d/volumes/kubernetes.io~projected/kube-api-access-r7vx6 DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc/userdata/shm DeviceMajor:0 DeviceMinor:65 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a5aa953f5d5148 MacAddress:5e:9a:03:66:60:2d Speed:10000 Mtu:8901} {Name:0c05f4b645ae4b2 MacAddress:9e:a5:f3:2e:e1:e3 Speed:10000 Mtu:8901} {Name:1542a79cb584e32 MacAddress:62:1a:0f:f0:77:2c Speed:10000 Mtu:8901} {Name:301eacfa12c11dc MacAddress:c2:87:7e:28:d9:02 Speed:10000 Mtu:8901} {Name:431cc09c53a3f91 MacAddress:ce:51:db:be:66:5d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:55:f1:39:e8:ad Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:dd:99:3b:32:73 Speed:0 Mtu:8901} {Name:c9bfdfe29b6796e MacAddress:32:d0:59:e8:3c:42 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:55:f1:39:e8:ad Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:12:60:db:34:ca Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:b6:57:25:1e:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431265    4493 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431324    4493 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431695    4493 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431721    4493 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-75.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431892    4493 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431900    4493 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431910    4493 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.431922    4493 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432014    4493 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432194    4493 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432346    4493 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432401    4493 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432417    4493 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432427    4493 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432436    4493 kubelet.go:405] "Adding apiserver pod source"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.432446    4493 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.434303    4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.434329    4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.435222    4493 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.446755    4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.446765    4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.446868    4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447077    4493 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447105    4493 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447250    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447263    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447269    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447275    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447282    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447292    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447301    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447317    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447327    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447334    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447342    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447352    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447377    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 00:23:30.448006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.447386    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 00:23:30.451210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.451196    4493 server.go:1263] "Started kubelet"
Nov 01 00:23:30.451324 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.451292    4493 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 00:23:30.451421 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.451379    4493 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 00:23:30.451472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.451445    4493 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 00:23:30.451751 ip-10-0-141-75 systemd[1]: Started Kubernetes Kubelet.
Nov 01 00:23:30.452415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.452371    4493 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 00:23:30.452527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.452511    4493 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 00:23:30.454230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454198    4493 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 00:23:30.454230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454224    4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454364    4493 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454376    4493 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454459    4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:51 +0000 UTC" deadline="2028-05-04 10:27:25.708493313 +0000 UTC"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.454471    4493 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454484    4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13210h3m55.254012889s"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.454520    4493 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455083    4493 factory.go:55] Registering systemd factory
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455100    4493 factory.go:223] Registration of the systemd container factory successfully
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455335    4493 factory.go:153] Registering CRI-O factory
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455346    4493 factory.go:223] Registration of the crio container factory successfully
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455403    4493 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455420    4493 factory.go:103] Registering Raw factory
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455439    4493 manager.go:1199] Started watching for new ooms in manager
Nov 01 00:23:30.456172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.455762    4493 manager.go:319] Starting recovery of all containers
Nov 01 00:23:30.456729 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456695    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051fe699-0a2b-4aca-83fa-a034cd8788af" volumeName="kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir" seLinuxMountContext=""
Nov 01 00:23:30.456780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456735    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a11cec8-d010-49d5-876c-fbd012e9da8d" volumeName="kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir" seLinuxMountContext=""
Nov 01 00:23:30.456780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456750    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fd5adf-add5-424c-b999-ec0c3b16374b" volumeName="kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw" seLinuxMountContext=""
Nov 01 00:23:30.456780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456762    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dbea131-444d-428f-ac3f-90b26919def3" volumeName="kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456787    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dbea131-444d-428f-ac3f-90b26919def3" volumeName="kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456800    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="966f1b0f-5b6a-4dc9-ac24-1ad5de31f629" volumeName="kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456817    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="966f1b0f-5b6a-4dc9-ac24-1ad5de31f629" volumeName="kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456830    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a67902a9-529f-4bbe-831b-1e381afd93d8" volumeName="kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456845    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fd5adf-add5-424c-b999-ec0c3b16374b" volumeName="kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456858    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dbea131-444d-428f-ac3f-90b26919def3" volumeName="kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key" seLinuxMountContext=""
Nov 01 00:23:30.456875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456870    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b2d432-059c-4433-89dd-f947cf07a66d" volumeName="kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456884    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b2d432-059c-4433-89dd-f947cf07a66d" volumeName="kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456897    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77f3059-b3bd-4359-95fc-2137bab8feef" volumeName="kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456913    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051fe699-0a2b-4aca-83fa-a034cd8788af" volumeName="kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456928    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27aafa36-67af-46cb-b144-46b6c60ffb4b" volumeName="kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456940    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3554c745-2c9d-43a8-a83e-50089377ab99" volumeName="kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456952    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b2d432-059c-4433-89dd-f947cf07a66d" volumeName="kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456965    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b2d432-059c-4433-89dd-f947cf07a66d" volumeName="kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456979    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77f3059-b3bd-4359-95fc-2137bab8feef" volumeName="kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.456992    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77f3059-b3bd-4359-95fc-2137bab8feef" volumeName="kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457005    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a11cec8-d010-49d5-876c-fbd012e9da8d" volumeName="kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457017    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27aafa36-67af-46cb-b144-46b6c60ffb4b" volumeName="kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457029    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77f3059-b3bd-4359-95fc-2137bab8feef" volumeName="kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457041    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373" volumeName="kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457055    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373" volumeName="kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457068    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27aafa36-67af-46cb-b144-46b6c60ffb4b" volumeName="kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457081    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="830a85cd-24d7-4745-8274-cf8e043df4d7" volumeName="kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r" seLinuxMountContext=""
Nov 01 00:23:30.457100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457094    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af83a55a-8226-460e-bc4f-ae16df4b4a33" volumeName="kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457111    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b2d432-059c-4433-89dd-f947cf07a66d" volumeName="kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457124    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f532669a-da34-4f31-bbac-a4ee3813e9a5" volumeName="kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457156    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59a4c1e-8bec-4377-bea2-c86c047de702" volumeName="kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457168    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd823994-2ff6-4e3b-b172-97f93135b0df" volumeName="kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457182    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051fe699-0a2b-4aca-83fa-a034cd8788af" volumeName="kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457195    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2affb2ab-ec09-43ea-8699-8654990567f3" volumeName="kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457207    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fd5adf-add5-424c-b999-ec0c3b16374b" volumeName="kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457219    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="813dbf2f-9f9e-4aed-bebe-caf642a1e7f3" volumeName="kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457233    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af83a55a-8226-460e-bc4f-ae16df4b4a33" volumeName="kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457246    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59a4c1e-8bec-4377-bea2-c86c047de702" volumeName="kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457258    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27aafa36-67af-46cb-b144-46b6c60ffb4b" volumeName="kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457270    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="966f1b0f-5b6a-4dc9-ac24-1ad5de31f629" volumeName="kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457281    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2affb2ab-ec09-43ea-8699-8654990567f3" volumeName="kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457293    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="830a85cd-24d7-4745-8274-cf8e043df4d7" volumeName="kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume" seLinuxMountContext=""
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457305    4493 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 00:23:30.457667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.457314    4493 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 00:23:30.493300 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.493206    4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 00:23:30.494818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.494797    4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 00:23:30.494818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.494822    4493 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 00:23:30.494994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.494842    4493 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 00:23:30.494994 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.494885    4493 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 00:23:30.528759 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.528722    4493 generic.go:356] "Generic (PLEG): container finished" podID="256267ec21e36d6165c8dc2e9e70ea9f" containerID="18d0504c721df0fa961099eec1fc07bf070c8046ab2676dfa18b0fb16e738909" exitCode=0
Nov 01 00:23:30.540440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.540418    4493 manager.go:324] Recovery completed
Nov 01 00:23:30.546498 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546458    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="014c708f2833255012e893de0f3bf0f29fae1b9d0d351c17ceb44121d7e09975" exitCode=0
Nov 01 00:23:30.546498 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546493    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="bee8d76163fc68ef4e9f65394b7d601affb4c9069873f576cade2c6b0dfd6342" exitCode=0
Nov 01 00:23:30.546498 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546503    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="9d519cc13ad590f32979f401a162949a82ec4d1eb4b033bf3423d288d9ca10e3" exitCode=0
Nov 01 00:23:30.546764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546510    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="964cf9431d60833d561588cbcc0dfe196c6da20828c966fbcf272b48f0e4fdac" exitCode=0
Nov 01 00:23:30.546764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546519    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="0249b96330f22440443e52ed9126c3c59292d1f8ec0d4805cea21dd83c6176a3" exitCode=0
Nov 01 00:23:30.546764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.546527    4493 generic.go:356] "Generic (PLEG): container finished" podID="d77f3059-b3bd-4359-95fc-2137bab8feef" containerID="4792ab451d95d922e484bf9872adf26cd909fc1d46b0743a1fdc0c941ecb12e6" exitCode=0
Nov 01 00:23:30.554553 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.554530    4493 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:23:30.555614 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.555600    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.556678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.556660    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.556753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.556697    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.556753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.556712    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.558476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558458    4493 cpu_manager.go:226] "Starting" policy="none"
Nov 01 00:23:30.558476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558475    4493 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 00:23:30.558677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558476    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.558677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558497    4493 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 00:23:30.558677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558644    4493 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 00:23:30.558677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558657    4493 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 00:23:30.558677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558677    4493 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558685    4493 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558693    4493 policy_none.go:50] "Start"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558706    4493 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558716    4493 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558860    4493 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558868    4493 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 00:23:30.558890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.558873    4493 policy_none.go:44] "Start"
Nov 01 00:23:30.559382 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.559366    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.559463 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.559395    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.559463 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.559411    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.563969 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.563925    4493 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 00:23:30.568361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568347    4493 manager.go:341] "Starting Device Plugin manager"
Nov 01 00:23:30.568483 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.568384    4493 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 00:23:30.568483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568397    4493 server.go:87] "Starting device plugin registration server"
Nov 01 00:23:30.568719 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568706    4493 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 00:23:30.568772 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568719    4493 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 00:23:30.568868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568854    4493 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 00:23:30.568977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568947    4493 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 00:23:30.569033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568986    4493 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 00:23:30.569033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.568992    4493 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 00:23:30.569147 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.569051    4493 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:23:30.568975922Z","UUID":"395c342b-0414-44d8-92c1-eabf9dc64d09","Handler":null,"Name":"","Endpoint":""}
Nov 01 00:23:30.571082 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.571058    4493 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 00:23:30.571199 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.571120    4493 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-75.us-west-1.compute.internal\" not found"
Nov 01 00:23:30.571710 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.571691    4493 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 00:23:30.571784 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.571730    4493 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 00:23:30.595008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.594965    4493 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"]
Nov 01 00:23:30.595172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.595101    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.596263 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.596245    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.596354 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.596276    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.596354 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.596287    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.596562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.596548    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.596700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.596684    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.597341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597323    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.597410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597351    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.597410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597366    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.597410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597374    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.597410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597390    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.597410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597403    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.597585 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.597524    4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.597725 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597677    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"e15a76296c7098967185b7bb6ae1aa66","Type":"ContainerStarted","Data":"d45fe9605ecb741a11400ae040abbac1afd7fb0b2f0fcd5cd04a9ce016cc7e2c"}
Nov 01 00:23:30.597782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597734    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"e15a76296c7098967185b7bb6ae1aa66","Type":"ContainerStarted","Data":"8cff5e59ac15453f33c42fa4274969784b0817cd21335f485c6eba8d6622c4cc"}
Nov 01 00:23:30.597782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597750    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerStarted","Data":"01941b1f9118fa7da1c37f5fdbe8ff624143b31600ad209e4157cac2824c47d5"}
Nov 01 00:23:30.597782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597762    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerDied","Data":"18d0504c721df0fa961099eec1fc07bf070c8046ab2676dfa18b0fb16e738909"}
Nov 01 00:23:30.597782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597775    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal" event={"ID":"256267ec21e36d6165c8dc2e9e70ea9f","Type":"ContainerStarted","Data":"a907102dcbd63b762138a354cd3165ca8a043408d9986e1d1a36bda022726f62"}
Nov 01 00:23:30.597959 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.597802    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.598890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.598873    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.598960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.598904    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.598960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.598917    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.599062 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.599049    4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-75.us-west-1.compute.internal\" not found" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.645257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.645225    4493 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:23:30.645447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.645285    4493 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:23:30.645447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.645421    4493 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:23:30.655087 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.655056    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.669282 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.669253    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 00:23:30.670761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.670739    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 00:23:30.670905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.670780    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 00:23:30.670905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.670795    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 00:23:30.670905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.670819    4493 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.696310 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.696280    4493 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 00:23:30.696468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.696378    4493 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 00:23:30.714016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.713985    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:23:30.714182 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.714046    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.714182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.714063    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.777818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.777752    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:23:30.777818 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:30.777804    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.796528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.796476    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.796584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.796529    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.796584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.796546    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.796584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.796572    4493 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.796680 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.796639    4493 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.797201 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.797187    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 00:23:30.797257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.797216    4493 setters.go:546] "Node became not ready" node="ip-10-0-141-75.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T00:23:30Z","lastTransitionTime":"2026-11-01T00:23:30Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 00:23:30.810559 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.810542    4493 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-75.us-west-1.compute.internal" is owned by different node (oldNodeID="89b45730-0311-4047-8e2b-aeca39cd1f2d", newNodeID=""), cleaning up...
Nov 01 00:23:30.897419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897385    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.897419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.897617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897440    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.897617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897491    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.897617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897506    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/256267ec21e36d6165c8dc2e9e70ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"256267ec21e36d6165c8dc2e9e70ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:30.897617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:30.897508    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15a76296c7098967185b7bb6ae1aa66-config\") pod \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" (UID: \"e15a76296c7098967185b7bb6ae1aa66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:31.433545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.433511    4493 apiserver.go:52] "Watching apiserver"
Nov 01 00:23:31.553502 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.553476    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:31.553678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.553589    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:31.607117 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.607088    4493 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 00:23:31.607555 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.607536    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gzc59","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g","openshift-dns/node-resolver-5d2nq","openshift-service-ca/service-ca-5fffc97bdf-mn6ww","openshift-multus/network-metrics-daemon-lc7vk","openshift-insights/insights-runtime-extractor-pd52b","openshift-multus/multus-additional-cni-plugins-xcdqw","openshift-dns/dns-default-d797j","openshift-image-registry/node-ca-qbp6l","kube-system/konnectivity-agent-hcfh5","openshift-ingress-canary/ingress-canary-qfkjh","kube-system/global-pull-secret-syncer-q456m","openshift-insights/periodic-gathering-h6sdm-nnstj","openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-6kbnk","openshift-multus/multus-qpbhj","openshift-network-diagnostics/network-check-target-ccss8","openshift-network-operator/iptables-alerter-vd2ph"]
Nov 01 00:23:31.612350 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.611973    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:31.613911 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.613893    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:23:31.614014 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.613935    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:31.614467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.614443    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 00:23:31.614562 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.614506    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-75.us-west-1.compute.internal"
Nov 01 00:23:31.615814 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.615797    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.616223 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.616208    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.656525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.656498    4493 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 00:23:31.661370 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.661347    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.661370 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.661349    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 00:23:31.661610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.661347    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 00:23:31.661610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.661399    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 00:23:31.661610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.661524    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\""
Nov 01 00:23:31.662541 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.662527    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\""
Nov 01 00:23:31.700797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700735    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.700797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.700797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700781    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700800    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vx6\" (UniqueName: \"kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700867    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.701004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700904    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.701004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700936    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllcf\" (UniqueName: \"kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.701004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.700961    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701078    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl4g\" (UniqueName: \"kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701123    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701176    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701202    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701246    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701270    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701301    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701328    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701352    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701377    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701493    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701519    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701593    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:31.701631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701616    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701656    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701672    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701750    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701775    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701795    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701816    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701854    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7w2j\" (UniqueName: \"kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701882    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:31.702008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.701982    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702024    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702058    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702076    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702092    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702109    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702124    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tsc\" (UniqueName: \"kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702128    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702156    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702194    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqs6\" (UniqueName: \"kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702260    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznhf\" (UniqueName: \"kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702307    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-tmp\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702352    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702480    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702508    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702526    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702566    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702599    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702628    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702652    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702682    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702699    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.702698 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702714    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702734    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702767    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702837    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702888    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkdh\" (UniqueName: \"kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702954    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.702987    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703011    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703040    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703076    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cni-binary-copy\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703090    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703097    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/830a85cd-24d7-4745-8274-cf8e043df4d7-data-volume\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703118    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703169    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703199    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703203    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fe699-0a2b-4aca-83fa-a034cd8788af-tmp-dir\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703239    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703257    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703274    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703293    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703322    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703354    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a11cec8-d010-49d5-876c-fbd012e9da8d-tmp-dir\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-tuned\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703378    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703416    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703449    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703470    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703493    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703507    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703562    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703592    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703613    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45zt\" (UniqueName: \"kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:31.703682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703653    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703724    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703748    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703769    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703807    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703839    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrj9r\" (UniqueName: \"kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703859    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703877    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.703975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704008    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704054    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704078    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704093    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704117    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704166    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704226    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnd8\" (UniqueName: \"kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8\") pod \"migrator-66df5d496b-54cjs\" (UID: \"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704317    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704443    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704467    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f92\" (UniqueName: \"kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704482    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704500    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8rl\" (UniqueName: \"kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.704600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704527    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.705209 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.704872    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77f3059-b3bd-4359-95fc-2137bab8feef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.766378 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.766331    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.766799 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.766725    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.766812    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 00:23:31.767045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.766940    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 00:23:31.767045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.766997    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.767329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767047    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 00:23:31.767329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767056    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 00:23:31.767329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767194    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.767329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767263    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\""
Nov 01 00:23:31.767553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767384    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\""
Nov 01 00:23:31.767553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767456    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.767553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767455    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\""
Nov 01 00:23:31.767553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767474    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767551    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 00:23:31.767726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767612    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 00:23:31.767726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767656    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\""
Nov 01 00:23:31.767726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767701    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 00:23:31.767878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767865    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.767925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767901    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767923    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767927    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767937    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.767970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767949    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 00:23:31.767970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767963    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.767928    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768005    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768014    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768014    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768058    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768161    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 00:23:31.768212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768176    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 00:23:31.768434 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768262    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.768942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.768930    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.771259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771239    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-env-overrides\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.771259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771248    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-daemon-config\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.771525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771507    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl4g\" (UniqueName: \"kubernetes.io/projected/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-kube-api-access-cxl4g\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.771643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771623    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vx6\" (UniqueName: \"kubernetes.io/projected/d0b2d432-059c-4433-89dd-f947cf07a66d-kube-api-access-r7vx6\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.771759 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771745    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dbea131-444d-428f-ac3f-90b26919def3-signing-cabundle\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.771815 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771765    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-config\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.771869 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771837    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.771869 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.771836    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2affb2ab-ec09-43ea-8699-8654990567f3-konnectivity-ca\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:31.772322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.772301    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgpw\" (UniqueName: \"kubernetes.io/projected/41fd5adf-add5-424c-b999-ec0c3b16374b-kube-api-access-gdgpw\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.772438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.772425    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0b2d432-059c-4433-89dd-f947cf07a66d-ovn-node-metrics-cert\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.772872 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.772851    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqs6\" (UniqueName: \"kubernetes.io/projected/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-kube-api-access-2kqs6\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:31.773034 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.773017    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/051fe699-0a2b-4aca-83fa-a034cd8788af-config-volume\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.773249 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.773233    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrj9r\" (UniqueName: \"kubernetes.io/projected/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-api-access-jrj9r\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.773803 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.773785    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45zt\" (UniqueName: \"kubernetes.io/projected/f532669a-da34-4f31-bbac-a4ee3813e9a5-kube-api-access-c45zt\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:31.773904 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.773887    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkdh\" (UniqueName: \"kubernetes.io/projected/a67902a9-529f-4bbe-831b-1e381afd93d8-kube-api-access-5qkdh\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.773947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.773904    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f92\" (UniqueName: \"kubernetes.io/projected/d77f3059-b3bd-4359-95fc-2137bab8feef-kube-api-access-l9f92\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.774818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.774802    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.774908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.774895    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dbea131-444d-428f-ac3f-90b26919def3-signing-key\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.776218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.776203    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnd8\" (UniqueName: \"kubernetes.io/projected/813dbf2f-9f9e-4aed-bebe-caf642a1e7f3-kube-api-access-vbnd8\") pod \"migrator-66df5d496b-54cjs\" (UID: \"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs"
Nov 01 00:23:31.776401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.776387    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8rl\" (UniqueName: \"kubernetes.io/projected/6dbea131-444d-428f-ac3f-90b26919def3-kube-api-access-pk8rl\") pod \"service-ca-5fffc97bdf-mn6ww\" (UID: \"6dbea131-444d-428f-ac3f-90b26919def3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww"
Nov 01 00:23:31.805747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805706    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805796    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805818    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805845    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-host\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805861    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.805916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-cnibin\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805921    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805917    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-multus\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805957    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805974    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-netns\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805981    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-var-lib-kubelet\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.805995    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806002    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-netns\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805977    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806038    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.806064    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.306047599 +0000 UTC m=+1.983634522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.805999    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-hostroot\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806080    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806064    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-var-lib-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806106    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806122    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806164    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806190    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806195    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-netd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806198    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-etc-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806212    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806249    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af83a55a-8226-460e-bc4f-ae16df4b4a33-host\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806265    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806289    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806304    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806321    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806344    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806269    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-socket-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806364    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-dbus\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806367    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806371    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-systemd-units\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806365    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-conf-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806439    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-conf\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806444    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806449    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-cni-bin\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806489    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-systemd\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806503    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806530    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-ovn\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-modprobe-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806604    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-registration-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806619    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-sys\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806620    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806685    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-socket-dir-parent\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806705    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806739    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806778    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806783    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-sys-fs\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806812    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806818    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.806836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806837    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806861    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806873    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-run-openvswitch\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806884    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806913    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-kubernetes\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806928    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806931    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77f3059-b3bd-4359-95fc-2137bab8feef-system-cni-dir\") pod \"multus-additional-cni-plugins-xcdqw\" (UID: \"d77f3059-b3bd-4359-95fc-2137bab8feef\") " pod="openshift-multus/multus-additional-cni-plugins-xcdqw"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806959    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-kubelet\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806980    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.806998    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-log-socket\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807305    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807359    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807379    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807390    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807435    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807464    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-run\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807536    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysconfig\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807545    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807614    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807713    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/830a85cd-24d7-4745-8274-cf8e043df4d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807731    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-sysctl-d\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807700    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f59a4c1e-8bec-4377-bea2-c86c047de702-host-slash\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807786    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807817    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807861    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807897    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-k8s-cni-cncf-io\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807966    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-slash\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.807900    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808008    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808030    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/830a85cd-24d7-4745-8274-cf8e043df4d7-crio-socket\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808087    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808160    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-run-multus-certs\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.808184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808194    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808223    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808271    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808310    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808338    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808369    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808400    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808427    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808900    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808932    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808961    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.808990    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809077    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a67902a9-529f-4bbe-831b-1e381afd93d8-device-dir\") pod \"aws-ebs-csi-driver-node-tnc5g\" (UID: \"a67902a9-529f-4bbe-831b-1e381afd93d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tnc5g"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809118    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-cni-bin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809194    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-node-log\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809255    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0b2d432-059c-4433-89dd-f947cf07a66d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809309    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a11cec8-d010-49d5-876c-fbd012e9da8d-hosts-file\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809361    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3554c745-2c9d-43a8-a83e-50089377ab99-kubelet-config\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809416    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-multus-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.809483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809470    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-cnibin\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.809557    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.809619    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.309595838 +0000 UTC m=+1.987182761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809674    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-etc-systemd\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809780    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41fd5adf-add5-424c-b999-ec0c3b16374b-lib-modules\") pod \"tuned-gzc59\" (UID: \"41fd5adf-add5-424c-b999-ec0c3b16374b\") " pod="openshift-cluster-node-tuning-operator/tuned-gzc59"
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809833    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-system-cni-dir\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809880    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-host-var-lib-kubelet\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.810435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.809971    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/966f1b0f-5b6a-4dc9-ac24-1ad5de31f629-etc-kubernetes\") pod \"multus-qpbhj\" (UID: \"966f1b0f-5b6a-4dc9-ac24-1ad5de31f629\") " pod="openshift-multus/multus-qpbhj"
Nov 01 00:23:31.810895 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.810871    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 00:23:31.811025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.810865    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.811025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.810907    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.811025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.810915    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811028    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811007    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811060    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811080    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811087    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811100    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.811197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811182    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\""
Nov 01 00:23:31.811544 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811531    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 00:23:31.811544 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.811538    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af83a55a-8226-460e-bc4f-ae16df4b4a33-serviceca\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.812065 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812047    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 00:23:31.812178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812160    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\""
Nov 01 00:23:31.812178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812167    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cvjqf\""
Nov 01 00:23:31.812178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812159    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\""
Nov 01 00:23:31.812285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812205    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2ck8t\""
Nov 01 00:23:31.812458 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812441    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tsc\" (UniqueName: \"kubernetes.io/projected/1a11cec8-d010-49d5-876c-fbd012e9da8d-kube-api-access-b6tsc\") pod \"node-resolver-5d2nq\" (UID: \"1a11cec8-d010-49d5-876c-fbd012e9da8d\") " pod="openshift-dns/node-resolver-5d2nq"
Nov 01 00:23:31.812500 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.812486    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznhf\" (UniqueName: \"kubernetes.io/projected/f59a4c1e-8bec-4377-bea2-c86c047de702-kube-api-access-bznhf\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.813478 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.813452    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 00:23:31.813478 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.813456    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-plg4w\""
Nov 01 00:23:31.813564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.813539    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\""
Nov 01 00:23:31.813564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.813540    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\""
Nov 01 00:23:31.813878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.813864    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9m9\" (UniqueName: \"kubernetes.io/projected/fd823994-2ff6-4e3b-b172-97f93135b0df-kube-api-access-lf9m9\") pod \"network-check-target-ccss8\" (UID: \"fd823994-2ff6-4e3b-b172-97f93135b0df\") " pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:31.815038 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.815025    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 00:23:31.815171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.815157    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\""
Nov 01 00:23:31.815212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.815175    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 00:23:31.815212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.815203    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\""
Nov 01 00:23:31.815974 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.815963    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 00:23:31.816291 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.816264    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 00:23:31.816341 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:31.816316    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.31630417 +0000 UTC m=+1.993891107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : secret "metrics-daemon-secret" not found
Nov 01 00:23:31.817573 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.817557    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 00:23:31.820930 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.820909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7w2j\" (UniqueName: \"kubernetes.io/projected/af83a55a-8226-460e-bc4f-ae16df4b4a33-kube-api-access-d7w2j\") pod \"node-ca-qbp6l\" (UID: \"af83a55a-8226-460e-bc4f-ae16df4b4a33\") " pod="openshift-image-registry/node-ca-qbp6l"
Nov 01 00:23:31.820995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.820984    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllcf\" (UniqueName: \"kubernetes.io/projected/051fe699-0a2b-4aca-83fa-a034cd8788af-kube-api-access-rllcf\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:31.821914 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.821894    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0b2d432-059c-4433-89dd-f947cf07a66d-ovnkube-script-lib\") pod \"ovnkube-node-6kbnk\" (UID: \"d0b2d432-059c-4433-89dd-f947cf07a66d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:31.821963 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.821941    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373-cert\") pod \"ingress-canary-qfkjh\" (UID: \"e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373\") " pod="openshift-ingress-canary/ingress-canary-qfkjh"
Nov 01 00:23:31.822235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.822222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2affb2ab-ec09-43ea-8699-8654990567f3-agent-certs\") pod \"konnectivity-agent-hcfh5\" (UID: \"2affb2ab-ec09-43ea-8699-8654990567f3\") " pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:31.823077 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.823051    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3554c745-2c9d-43a8-a83e-50089377ab99-original-pull-secret\") pod \"global-pull-secret-syncer-q456m\" (UID: \"3554c745-2c9d-43a8-a83e-50089377ab99\") " pod="kube-system/global-pull-secret-syncer-q456m"
Nov 01 00:23:31.824153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.824123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f59a4c1e-8bec-4377-bea2-c86c047de702-iptables-alerter-script\") pod \"iptables-alerter-vd2ph\" (UID: \"f59a4c1e-8bec-4377-bea2-c86c047de702\") " pod="openshift-network-operator/iptables-alerter-vd2ph"
Nov 01 00:23:31.824346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.824328    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"periodic-gathering-h6sdm-nnstj\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") " pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:23:31.918356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:31.918338    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:23:32.212588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:32.212563    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:32.313068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:32.313028    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:32.313068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:32.313072    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:32.313294 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.313182    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:32.313294 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.313203    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:32.313294 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.313259    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.313239912 +0000 UTC m=+2.990826835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:32.313294 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.313272    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.313266203 +0000 UTC m=+2.990853126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:32.414465 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:32.414434    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:32.414619 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.414539    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 00:23:32.414619 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:32.414590    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.414577999 +0000 UTC m=+3.092164922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : secret "metrics-daemon-secret" not found
Nov 01 00:23:33.321931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:33.321888    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:33.322352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:33.321947    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:33.322352 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.322032    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:33.322352 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.322091    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:33.322352 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.322096    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.322081109 +0000 UTC m=+4.999668045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:33.322352 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.322150    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.322120942 +0000 UTC m=+4.999707865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:33.422746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:33.422665    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:33.422885 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.422791    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 00:23:33.422885 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:33.422856    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.422836753 +0000 UTC m=+5.100423689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : secret "metrics-daemon-secret" not found
Nov 01 00:23:33.559814 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:33.559785    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-54cjs" event={"ID":"813dbf2f-9f9e-4aed-bebe-caf642a1e7f3","Type":"ContainerStarted","Data":"51af0217151290088f849d2459359fd37e5897e71a1e58dc15279c6d7fe1af78"}
Nov 01 00:23:33.561286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:33.561262    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h6sdm-nnstj" event={"ID":"27aafa36-67af-46cb-b144-46b6c60ffb4b","Type":"ContainerStarted","Data":"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"}
Nov 01 00:23:34.228722 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:34.228681    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:34.228988 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:34.228929    4493 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 00:23:35.337471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:35.337433    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:35.337471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:35.337472    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:35.337890 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.337578    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 00:23:35.337890 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.337579    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:35.337890 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.337630    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls podName:051fe699-0a2b-4aca-83fa-a034cd8788af nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.337618742 +0000 UTC m=+9.015205679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls") pod "dns-default-d797j" (UID: "051fe699-0a2b-4aca-83fa-a034cd8788af") : secret "dns-default-metrics-tls" not found
Nov 01 00:23:35.337890 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.337643    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls podName:830a85cd-24d7-4745-8274-cf8e043df4d7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.337637913 +0000 UTC m=+9.015224836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd52b" (UID: "830a85cd-24d7-4745-8274-cf8e043df4d7") : secret "insights-runtime-extractor-tls" not found
Nov 01 00:23:35.438664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:35.438625    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:35.438827 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.438728    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 00:23:35.438827 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:35.438781    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs podName:f532669a-da34-4f31-bbac-a4ee3813e9a5 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.43876753 +0000 UTC m=+9.116354465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs") pod "network-metrics-daemon-lc7vk" (UID: "f532669a-da34-4f31-bbac-a4ee3813e9a5") : secret "metrics-daemon-secret" not found
Nov 01 00:23:35.568384 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:35.568357    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mn6ww" event={"ID":"6dbea131-444d-428f-ac3f-90b26919def3","Type":"ContainerStarted","Data":"a5774cec640f684fac74398e7669cabb8de564c9badd8f47c07d370b45228870"}
Nov 01 00:23:36.838307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:36.838282    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5d2nq_1a11cec8-d010-49d5-876c-fbd012e9da8d/dns-node-resolver/0.log"
Nov 01 00:23:37.888259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:37.888236    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kbnk"
Nov 01 00:23:38.031226 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:38.031196    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qbp6l_af83a55a-8226-460e-bc4f-ae16df4b4a33/node-ca/0.log"
Nov 01 00:23:38.832265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:38.832232    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qfkjh_e56ff68b-cfa4-4bdd-b1ea-ab0d1fd5c373/serve-healthcheck-canary/0.log"
Nov 01 00:23:39.032460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.032432    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-54cjs_813dbf2f-9f9e-4aed-bebe-caf642a1e7f3/migrator/0.log"
Nov 01 00:23:39.368450 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.368408    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:39.368631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.368458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:39.368880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.368861    4493 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 00:23:39.371237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.371212    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/051fe699-0a2b-4aca-83fa-a034cd8788af-metrics-tls\") pod \"dns-default-d797j\" (UID: \"051fe699-0a2b-4aca-83fa-a034cd8788af\") " pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:39.371357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.371212    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/830a85cd-24d7-4745-8274-cf8e043df4d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd52b\" (UID: \"830a85cd-24d7-4745-8274-cf8e043df4d7\") " pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:39.416440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.416409    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:39.416615 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.416566    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd52b"
Nov 01 00:23:39.472531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.472495    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:39.478400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.478370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f532669a-da34-4f31-bbac-a4ee3813e9a5-metrics-certs\") pod \"network-metrics-daemon-lc7vk\" (UID: \"f532669a-da34-4f31-bbac-a4ee3813e9a5\") " pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:39.712664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.712581    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc7vk"
Nov 01 00:23:39.843568 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.843537    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d797j"]
Nov 01 00:23:39.846731 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:39.846706    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051fe699_0a2b_4aca_83fa_a034cd8788af.slice/crio-d86bdd62fce9b2a3677a7e871a953dd759239ec84bcfbd9f9fa6880f7f9c59ae WatchSource:0}: Error finding container d86bdd62fce9b2a3677a7e871a953dd759239ec84bcfbd9f9fa6880f7f9c59ae: Status 404 returned error can't find the container with id d86bdd62fce9b2a3677a7e871a953dd759239ec84bcfbd9f9fa6880f7f9c59ae
Nov 01 00:23:39.893431 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:39.893404    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd52b"]
Nov 01 00:23:39.896769 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:39.896730    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830a85cd_24d7_4745_8274_cf8e043df4d7.slice/crio-5471ce42f7708dd244efbc231a22b2c449bf6e5de423c0d07c34cd6844a35aa5 WatchSource:0}: Error finding container 5471ce42f7708dd244efbc231a22b2c449bf6e5de423c0d07c34cd6844a35aa5: Status 404 returned error can't find the container with id 5471ce42f7708dd244efbc231a22b2c449bf6e5de423c0d07c34cd6844a35aa5
Nov 01 00:23:40.082369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.082326    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc7vk"]
Nov 01 00:23:40.085968 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:40.085933    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf532669a_da34_4f31_bbac_a4ee3813e9a5.slice/crio-254e9a8e6aa220c4e4ab39e10828bdd610ab344950a3300a23f6b8a5c7ae8929 WatchSource:0}: Error finding container 254e9a8e6aa220c4e4ab39e10828bdd610ab344950a3300a23f6b8a5c7ae8929: Status 404 returned error can't find the container with id 254e9a8e6aa220c4e4ab39e10828bdd610ab344950a3300a23f6b8a5c7ae8929
Nov 01 00:23:40.265291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.265255    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:23:40.582097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.582061    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7vk" event={"ID":"f532669a-da34-4f31-bbac-a4ee3813e9a5","Type":"ContainerStarted","Data":"254e9a8e6aa220c4e4ab39e10828bdd610ab344950a3300a23f6b8a5c7ae8929"}
Nov 01 00:23:40.583641 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.583613    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd52b" event={"ID":"830a85cd-24d7-4745-8274-cf8e043df4d7","Type":"ContainerStarted","Data":"7a8b531b2ee4a3a4fca08b93b07923841ba3f4afa1b5f8a2490afaf3468dd28b"}
Nov 01 00:23:40.583767 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.583671    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd52b" event={"ID":"830a85cd-24d7-4745-8274-cf8e043df4d7","Type":"ContainerStarted","Data":"5471ce42f7708dd244efbc231a22b2c449bf6e5de423c0d07c34cd6844a35aa5"}
Nov 01 00:23:40.584675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:40.584653    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d797j" event={"ID":"051fe699-0a2b-4aca-83fa-a034cd8788af","Type":"ContainerStarted","Data":"d86bdd62fce9b2a3677a7e871a953dd759239ec84bcfbd9f9fa6880f7f9c59ae"}
Nov 01 00:23:41.001097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:41.001065    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-75.us-west-1.compute.internal" event="NodeReady"
Nov 01 00:23:41.589688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:41.589656    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd52b" event={"ID":"830a85cd-24d7-4745-8274-cf8e043df4d7","Type":"ContainerStarted","Data":"7a86b7facb22e23e5a15427266b60055d97a6e059283e0d55530f662df894d71"}
Nov 01 00:23:42.596082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:42.595399    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7vk" event={"ID":"f532669a-da34-4f31-bbac-a4ee3813e9a5","Type":"ContainerStarted","Data":"324311ebc9001c464fb422aaab37387ea508fa5185fa4567c77967b45dd76be7"}
Nov 01 00:23:42.596082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:42.595469    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc7vk" event={"ID":"f532669a-da34-4f31-bbac-a4ee3813e9a5","Type":"ContainerStarted","Data":"b3c15a91af3a34acfefde3c15454e261bc5b3f5b684596869a12f4b74ad67c7c"}
Nov 01 00:23:42.598228 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:42.598203    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d797j" event={"ID":"051fe699-0a2b-4aca-83fa-a034cd8788af","Type":"ContainerStarted","Data":"0d960801302e3fa45c12f450f83729c211680340177080fd5712bd6348266f4d"}
Nov 01 00:23:42.598373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:42.598247    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d797j" event={"ID":"051fe699-0a2b-4aca-83fa-a034cd8788af","Type":"ContainerStarted","Data":"40462455bf4d409eaeafe33d408d2b34aa61c56163de4edf704461572f4c0731"}
Nov 01 00:23:42.598373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:42.598277    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:43.534714 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:43.534688    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hcfh5"
Nov 01 00:23:43.603777 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:43.603747    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd52b" event={"ID":"830a85cd-24d7-4745-8274-cf8e043df4d7","Type":"ContainerStarted","Data":"53c0b712e4a5e45003a67fe4e2f02c0769eeda35f0780e9a1b36525c0ca2f737"}
Nov 01 00:23:44.988126 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:23:44.988091    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:23:46.704719 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.704686    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"]
Nov 01 00:23:46.706428 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.706412    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:46.762980 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.762955    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 00:23:46.763242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.763225    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6kglx\""
Nov 01 00:23:46.813108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.813074    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"]
Nov 01 00:23:46.813713 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.813679    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-89tqd"]
Nov 01 00:23:46.815676 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.815657    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7774bb8b6f-rr7rs"]
Nov 01 00:23:46.815820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.815803    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:46.817427 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.817410    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.830040 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.830018    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f15387e-7a26-4540-ab83-9d7dc0e9a650-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-phszk\" (UID: \"3f15387e-7a26-4540-ab83-9d7dc0e9a650\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:46.868066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.868034    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 00:23:46.868066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.868050    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 00:23:46.868359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.868345    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-md2tg\""
Nov 01 00:23:46.871455 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.871440    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9sbx\""
Nov 01 00:23:46.925535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.925506    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 00:23:46.926223 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.926211    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-89tqd"]
Nov 01 00:23:46.926795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.926777    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774bb8b6f-rr7rs"]
Nov 01 00:23:46.930935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.930918    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f15387e-7a26-4540-ab83-9d7dc0e9a650-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-phszk\" (UID: \"3f15387e-7a26-4540-ab83-9d7dc0e9a650\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:46.930997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.930946    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4cef8cb-a4b9-4738-9585-8b96f0f36525-ca-trust-extracted\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.930997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.930964    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6176a253-226f-4774-8804-9fce7f7d92ba-tmp\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:46.930997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.930982    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhd94\" (UniqueName: \"kubernetes.io/projected/6176a253-226f-4774-8804-9fce7f7d92ba-kube-api-access-lhd94\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:46.931111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931000    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-tls\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931079    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-certificates\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931223 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931206    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-image-registry-private-configuration\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931296    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-bound-sa-token\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931323    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hcr\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-kube-api-access-p2hcr\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931347    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-trusted-ca\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.931432 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.931397    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-installation-pull-secrets\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:46.933625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.933607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f15387e-7a26-4540-ab83-9d7dc0e9a650-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-phszk\" (UID: \"3f15387e-7a26-4540-ab83-9d7dc0e9a650\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:46.968861 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.968834    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 00:23:46.968994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.968875    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 00:23:46.968994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:46.968892    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 00:23:47.016675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.016646    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:47.032377 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032342    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-bound-sa-token\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032376    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hcr\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-kube-api-access-p2hcr\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032399    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-trusted-ca\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032425    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-installation-pull-secrets\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032476    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4cef8cb-a4b9-4738-9585-8b96f0f36525-ca-trust-extracted\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032502    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6176a253-226f-4774-8804-9fce7f7d92ba-tmp\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:47.032717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032545    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhd94\" (UniqueName: \"kubernetes.io/projected/6176a253-226f-4774-8804-9fce7f7d92ba-kube-api-access-lhd94\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:47.032717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032566    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-tls\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032595    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-certificates\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.032717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.032629    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-image-registry-private-configuration\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.033098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.033057    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6176a253-226f-4774-8804-9fce7f7d92ba-tmp\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:47.033537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.033514    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4cef8cb-a4b9-4738-9585-8b96f0f36525-ca-trust-extracted\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.033729 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.033601    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-trusted-ca\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.034346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.034323    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-certificates\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.036030 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.035992    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-image-registry-private-configuration\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.036253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.036235    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-registry-tls\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.036345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.036329    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4cef8cb-a4b9-4738-9585-8b96f0f36525-installation-pull-secrets\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.093387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.093358    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-bound-sa-token\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.094121 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.094094    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hcr\" (UniqueName: \"kubernetes.io/projected/d4cef8cb-a4b9-4738-9585-8b96f0f36525-kube-api-access-p2hcr\") pod \"image-registry-7774bb8b6f-rr7rs\" (UID: \"d4cef8cb-a4b9-4738-9585-8b96f0f36525\") " pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.095429 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.095411    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhd94\" (UniqueName: \"kubernetes.io/projected/6176a253-226f-4774-8804-9fce7f7d92ba-kube-api-access-lhd94\") pod \"downloads-6dd789c8b7-89tqd\" (UID: \"6176a253-226f-4774-8804-9fce7f7d92ba\") " pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:47.127469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.127440    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:23:47.132885 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.132870    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.387454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.387361    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"]
Nov 01 00:23:47.391413 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:47.391386    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f15387e_7a26_4540_ab83_9d7dc0e9a650.slice/crio-ad78f9c47b07f08b756e2c6b0c3892ddce719721b84ac5d00a486eec5d3d3927 WatchSource:0}: Error finding container ad78f9c47b07f08b756e2c6b0c3892ddce719721b84ac5d00a486eec5d3d3927: Status 404 returned error can't find the container with id ad78f9c47b07f08b756e2c6b0c3892ddce719721b84ac5d00a486eec5d3d3927
Nov 01 00:23:47.497393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.497366    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-89tqd"]
Nov 01 00:23:47.500541 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:47.500515    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6176a253_226f_4774_8804_9fce7f7d92ba.slice/crio-6628bb9c0e856558a3ba11ac1dc240627e3407b1a6e0db3122bfd4a1bda81587 WatchSource:0}: Error finding container 6628bb9c0e856558a3ba11ac1dc240627e3407b1a6e0db3122bfd4a1bda81587: Status 404 returned error can't find the container with id 6628bb9c0e856558a3ba11ac1dc240627e3407b1a6e0db3122bfd4a1bda81587
Nov 01 00:23:47.551650 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.551620    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774bb8b6f-rr7rs"]
Nov 01 00:23:47.554382 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:47.554358    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cef8cb_a4b9_4738_9585_8b96f0f36525.slice/crio-e0fa1360dd10015639a92a3f1112cadae355e0a80e1e149865c448c80c7eef78 WatchSource:0}: Error finding container e0fa1360dd10015639a92a3f1112cadae355e0a80e1e149865c448c80c7eef78: Status 404 returned error can't find the container with id e0fa1360dd10015639a92a3f1112cadae355e0a80e1e149865c448c80c7eef78
Nov 01 00:23:47.617837 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.617808    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs" event={"ID":"d4cef8cb-a4b9-4738-9585-8b96f0f36525","Type":"ContainerStarted","Data":"b70d6d301e8c97eaf9ba9b1a759ac7f04761ff0bd64c681712a9af13e1aac876"}
Nov 01 00:23:47.617979 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.617863    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs" event={"ID":"d4cef8cb-a4b9-4738-9585-8b96f0f36525","Type":"ContainerStarted","Data":"e0fa1360dd10015639a92a3f1112cadae355e0a80e1e149865c448c80c7eef78"}
Nov 01 00:23:47.617979 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.617888    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:23:47.619175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.619155    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-89tqd" event={"ID":"6176a253-226f-4774-8804-9fce7f7d92ba","Type":"ContainerStarted","Data":"6628bb9c0e856558a3ba11ac1dc240627e3407b1a6e0db3122bfd4a1bda81587"}
Nov 01 00:23:47.620319 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.620301    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk" event={"ID":"3f15387e-7a26-4540-ab83-9d7dc0e9a650","Type":"ContainerStarted","Data":"ad78f9c47b07f08b756e2c6b0c3892ddce719721b84ac5d00a486eec5d3d3927"}
Nov 01 00:23:47.742659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:47.738817    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs" podStartSLOduration=1.7387994679999998 podStartE2EDuration="1.738799468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:47.73722861 +0000 UTC m=+17.414815555" watchObservedRunningTime="2026-11-01 00:23:47.738799468 +0000 UTC m=+17.416386414"
Nov 01 00:23:49.629444 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:49.629402    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk" event={"ID":"3f15387e-7a26-4540-ab83-9d7dc0e9a650","Type":"ContainerStarted","Data":"23592e9cd9b125fd1b41646ceaac61bf2640e12e00ef6f03bcbf99c819f9318c"}
Nov 01 00:23:49.629948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:49.629616    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:49.635283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:49.635253    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk"
Nov 01 00:23:49.744325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:49.744250    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-phszk" podStartSLOduration=2.3410035799999998 podStartE2EDuration="3.744231584s" totalImagesPullingTime="1.403228004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:49.74416361 +0000 UTC m=+19.421750553" watchObservedRunningTime="2026-11-01 00:23:49.744231584 +0000 UTC m=+19.421818529"
Nov 01 00:23:52.606055 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:52.606022    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d797j"
Nov 01 00:23:57.267499 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.267469    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xpmn4"]
Nov 01 00:23:57.270531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.270509    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.323117 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.323076    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 00:23:57.372954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.372918    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 00:23:57.372954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.372947    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 00:23:57.373171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.373002    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 00:23:57.373171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.373018    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 00:23:57.373171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.373058    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 00:23:57.373171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.373099    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vngfn\""
Nov 01 00:23:57.404357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404325    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404367    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6nh7\" (UniqueName: \"kubernetes.io/projected/2aef2bdc-97cd-4716-887e-b7756b67aeca-kube-api-access-t6nh7\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404447    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404485    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-root\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404506    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-sys\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404528    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-tls\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404759 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404578    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-textfile\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404759 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404601    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-wtmp\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.404829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.404753    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-metrics-client-ca\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.505961    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-root\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506004    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-sys\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506026    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-tls\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506061    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-textfile\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506085    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-wtmp\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506106    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-metrics-client-ca\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-root\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506127    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-sys\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6nh7\" (UniqueName: \"kubernetes.io/projected/2aef2bdc-97cd-4716-887e-b7756b67aeca-kube-api-access-t6nh7\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506217    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.506695 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.506483    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-textfile\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.507106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.507008    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-metrics-client-ca\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.507106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.507063    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.507284 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.507165    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-wtmp\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.508847 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.508828    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.509021 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.509000    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2aef2bdc-97cd-4716-887e-b7756b67aeca-node-exporter-tls\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.568165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.568078    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6nh7\" (UniqueName: \"kubernetes.io/projected/2aef2bdc-97cd-4716-887e-b7756b67aeca-kube-api-access-t6nh7\") pod \"node-exporter-xpmn4\" (UID: \"2aef2bdc-97cd-4716-887e-b7756b67aeca\") " pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.587049 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.587019    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpmn4"
Nov 01 00:23:57.598419 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:23:57.598389    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aef2bdc_97cd_4716_887e_b7756b67aeca.slice/crio-9896771300bc2e5b85742c8019df97cb2734af76d2f57f00aa9dbff85e8bd72d WatchSource:0}: Error finding container 9896771300bc2e5b85742c8019df97cb2734af76d2f57f00aa9dbff85e8bd72d: Status 404 returned error can't find the container with id 9896771300bc2e5b85742c8019df97cb2734af76d2f57f00aa9dbff85e8bd72d
Nov 01 00:23:57.660321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:57.660285    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpmn4" event={"ID":"2aef2bdc-97cd-4716-887e-b7756b67aeca","Type":"ContainerStarted","Data":"9896771300bc2e5b85742c8019df97cb2734af76d2f57f00aa9dbff85e8bd72d"}
Nov 01 00:23:59.669330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:59.669297    4493 generic.go:356] "Generic (PLEG): container finished" podID="2aef2bdc-97cd-4716-887e-b7756b67aeca" containerID="85172061eb9bf1e0cae8fa520b7eded8758d9113d8923fc5616248f3ff7e61a1" exitCode=0
Nov 01 00:23:59.669781 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:23:59.669364    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpmn4" event={"ID":"2aef2bdc-97cd-4716-887e-b7756b67aeca","Type":"ContainerDied","Data":"85172061eb9bf1e0cae8fa520b7eded8758d9113d8923fc5616248f3ff7e61a1"}
Nov 01 00:24:00.268431 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:00.268396    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ccss8"
Nov 01 00:24:00.676530 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:00.676441    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpmn4" event={"ID":"2aef2bdc-97cd-4716-887e-b7756b67aeca","Type":"ContainerStarted","Data":"70cc8fe69553ca056541f74e9ff4eedfe3504343934cfeb8e599a64f3776d961"}
Nov 01 00:24:00.676530 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:00.676527    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpmn4" event={"ID":"2aef2bdc-97cd-4716-887e-b7756b67aeca","Type":"ContainerStarted","Data":"aa784a905dbd69956284e5dc884d790af398a17cac1d2c460070bf2f1e1a7a88"}
Nov 01 00:24:00.845716 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:00.845658    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xpmn4" podStartSLOduration=2.388894826 podStartE2EDuration="3.845643327s" totalImagesPullingTime="1.372544604s" totalInitContainerRuntime="84.203897ms" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:00.845361113 +0000 UTC m=+30.522948058" watchObservedRunningTime="2026-11-01 00:24:00.845643327 +0000 UTC m=+30.523230272"
Nov 01 00:24:05.695977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:05.695928    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-89tqd" event={"ID":"6176a253-226f-4774-8804-9fce7f7d92ba","Type":"ContainerStarted","Data":"4a845778ea13e9ba2b61d82c47c59357a65efdd19f0b3a1e9d81efe6e3454b52"}
Nov 01 00:24:05.696936 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:05.696915    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:24:05.709249 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:05.709220    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-89tqd"
Nov 01 00:24:05.914638 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:05.914588    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-89tqd" podStartSLOduration=2.670518294 podStartE2EDuration="19.914573025s" totalImagesPullingTime="17.244054731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:05.913547052 +0000 UTC m=+35.591134012" watchObservedRunningTime="2026-11-01 00:24:05.914573025 +0000 UTC m=+35.592159964"
Nov 01 00:24:08.629167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:08.629118    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7774bb8b6f-rr7rs"
Nov 01 00:24:14.988635 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:24:14.988543    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:24:15.125401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:15.125372    4493 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hcfh5_2affb2ab-ec09-43ea-8699-8654990567f3/konnectivity-agent/0.log" limit=1024
Nov 01 00:24:44.989003 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:24:44.988958    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:24:47.507792 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.507751    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:24:47.508205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.508029    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-h6sdm-nnstj" podUID="27aafa36-67af-46cb-b144-46b6c60ffb4b" containerName="insights-gathering" containerID="cri-o://a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b" gracePeriod=30
Nov 01 00:24:47.779424 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.779363    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:24:47.829397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.829365    4493 generic.go:356] "Generic (PLEG): container finished" podID="27aafa36-67af-46cb-b144-46b6c60ffb4b" containerID="a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b" exitCode=2
Nov 01 00:24:47.829556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.829409    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h6sdm-nnstj"
Nov 01 00:24:47.829556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.829430    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h6sdm-nnstj" event={"ID":"27aafa36-67af-46cb-b144-46b6c60ffb4b","Type":"ContainerDied","Data":"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"}
Nov 01 00:24:47.829556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.829468    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h6sdm-nnstj" event={"ID":"27aafa36-67af-46cb-b144-46b6c60ffb4b","Type":"ContainerDied","Data":"301eacfa12c11dcdd139ade15dc3bece60d6ac5f6259df957312df3a2b792189"}
Nov 01 00:24:47.829556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.829497    4493 scope.go:122] "RemoveContainer" containerID="a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"
Nov 01 00:24:47.837642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.837626    4493 scope.go:122] "RemoveContainer" containerID="a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"
Nov 01 00:24:47.837919 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:24:47.837899    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b\": container with ID starting with a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b not found: ID does not exist" containerID="a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"
Nov 01 00:24:47.837989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.837928    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b"} err="failed to get container status \"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b\": rpc error: code = NotFound desc = could not find container \"a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b\": container with ID starting with a172aa7762516402f6ab2d5527ee43f30d44f35d854f5f46281b1e9c06ddf27b not found: ID does not exist"
Nov 01 00:24:47.876726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.876704    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") pod \"27aafa36-67af-46cb-b144-46b6c60ffb4b\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") "
Nov 01 00:24:47.876836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.876750    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") pod \"27aafa36-67af-46cb-b144-46b6c60ffb4b\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") "
Nov 01 00:24:47.876836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.876790    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") pod \"27aafa36-67af-46cb-b144-46b6c60ffb4b\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") "
Nov 01 00:24:47.876836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.876809    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") pod \"27aafa36-67af-46cb-b144-46b6c60ffb4b\" (UID: \"27aafa36-67af-46cb-b144-46b6c60ffb4b\") "
Nov 01 00:24:47.877175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.877128    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle" pod "27aafa36-67af-46cb-b144-46b6c60ffb4b" (UID: "27aafa36-67af-46cb-b144-46b6c60ffb4b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:24:47.877247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.877153    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path" pod "27aafa36-67af-46cb-b144-46b6c60ffb4b" (UID: "27aafa36-67af-46cb-b144-46b6c60ffb4b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:24:47.877247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.877189    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle" pod "27aafa36-67af-46cb-b144-46b6c60ffb4b" (UID: "27aafa36-67af-46cb-b144-46b6c60ffb4b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:24:47.879062 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.879042    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r" pod "27aafa36-67af-46cb-b144-46b6c60ffb4b" (UID: "27aafa36-67af-46cb-b144-46b6c60ffb4b"). InnerVolumeSpecName "kube-api-access-26m4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:24:47.978171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.978119    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26m4r\" (UniqueName: \"kubernetes.io/projected/27aafa36-67af-46cb-b144-46b6c60ffb4b-kube-api-access-26m4r\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:24:47.978171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.978163    4493 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-trusted-ca-bundle\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:24:47.978171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.978175    4493 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27aafa36-67af-46cb-b144-46b6c60ffb4b-archives-path\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:24:47.978402 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:47.978183    4493 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27aafa36-67af-46cb-b144-46b6c60ffb4b-service-ca-bundle\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:24:48.247990 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:48.247949    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:24:48.251651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:48.251628    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-h6sdm-nnstj"]
Nov 01 00:24:48.500072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:48.500042    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27aafa36-67af-46cb-b144-46b6c60ffb4b" path="/var/lib/kubelet/pods/27aafa36-67af-46cb-b144-46b6c60ffb4b/volumes"
Nov 01 00:24:56.682738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.682706    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"]
Nov 01 00:24:56.683356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.683073    4493 cpu_manager.go:414] "containerMap: removing container" podUID="27aafa36-67af-46cb-b144-46b6c60ffb4b" containerName="insights-gathering"
Nov 01 00:24:56.683356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.683089    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27aafa36-67af-46cb-b144-46b6c60ffb4b" containerName="insights-gathering"
Nov 01 00:24:56.683356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.683191    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="27aafa36-67af-46cb-b144-46b6c60ffb4b" containerName="insights-gathering"
Nov 01 00:24:56.685536 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.685515    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.740757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.740690    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 00:24:56.740757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.740706    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 00:24:56.740943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.740710    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 00:24:56.740943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.740792    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 00:24:56.791037 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.791005    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-76wlc\""
Nov 01 00:24:56.791228 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.791080    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 00:24:56.839355 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839324    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-telemeter-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839355 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839360    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839383    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-serving-certs-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839528    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd54d\" (UniqueName: \"kubernetes.io/projected/61859786-7308-482c-8959-38741e6a97ad-kube-api-access-cd54d\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839565    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-metrics-client-ca\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839623    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839643    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-federate-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.839755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.839699    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.891645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.891610    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 00:24:56.892634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.892616    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"]
Nov 01 00:24:56.941006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.940944    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.940982    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-federate-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941005    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941029    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-telemeter-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941162    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941205    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-serving-certs-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941256    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd54d\" (UniqueName: \"kubernetes.io/projected/61859786-7308-482c-8959-38741e6a97ad-kube-api-access-cd54d\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.941347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.941298    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-metrics-client-ca\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.942123 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.942063    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-metrics-client-ca\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.942271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.942255    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.942338 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.942265    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61859786-7308-482c-8959-38741e6a97ad-serving-certs-ca-bundle\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.943795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.943773    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-telemeter-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.943893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.943812    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.944292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.944273    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:56.944292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:56.944285    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/61859786-7308-482c-8959-38741e6a97ad-federate-client-tls\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:57.002163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:57.002117    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd54d\" (UniqueName: \"kubernetes.io/projected/61859786-7308-482c-8959-38741e6a97ad-kube-api-access-cd54d\") pod \"telemeter-client-5c4d9d98cb-czmcl\" (UID: \"61859786-7308-482c-8959-38741e6a97ad\") " pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:57.301867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:57.301834    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"
Nov 01 00:24:57.775461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:57.775432    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl"]
Nov 01 00:24:57.778495 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:24:57.778466    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61859786_7308_482c_8959_38741e6a97ad.slice/crio-8896e1c286b7a1af879d2b23deb1a056fb822dd3a659d1fa635aa02893d5158c WatchSource:0}: Error finding container 8896e1c286b7a1af879d2b23deb1a056fb822dd3a659d1fa635aa02893d5158c: Status 404 returned error can't find the container with id 8896e1c286b7a1af879d2b23deb1a056fb822dd3a659d1fa635aa02893d5158c
Nov 01 00:24:57.864495 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:57.864463    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl" event={"ID":"61859786-7308-482c-8959-38741e6a97ad","Type":"ContainerStarted","Data":"8896e1c286b7a1af879d2b23deb1a056fb822dd3a659d1fa635aa02893d5158c"}
Nov 01 00:24:59.872342 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:24:59.872304    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl" event={"ID":"61859786-7308-482c-8959-38741e6a97ad","Type":"ContainerStarted","Data":"b4cbc7c6ac9375c695011197a9ba43b9672649ca107e2a860176cf436b5044a4"}
Nov 01 00:25:01.881325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:25:01.881286    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl" event={"ID":"61859786-7308-482c-8959-38741e6a97ad","Type":"ContainerStarted","Data":"65bf1653b64cdaed94cfa6436d9d95aff87479159904cdcaae7e559f3a73d26a"}
Nov 01 00:25:01.881725 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:25:01.881345    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl" event={"ID":"61859786-7308-482c-8959-38741e6a97ad","Type":"ContainerStarted","Data":"8ac748f72523f9bf1a8c1e7d17798e61d032b462d5f83a8f0a648495bb89705f"}
Nov 01 00:25:02.104184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:25:02.104109    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c4d9d98cb-czmcl" podStartSLOduration=2.613054209 podStartE2EDuration="6.104095479s" totalImagesPullingTime="3.49104127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:25:02.053826305 +0000 UTC m=+91.731413250" watchObservedRunningTime="2026-11-01 00:25:02.104095479 +0000 UTC m=+91.781682448"
Nov 01 00:25:14.988658 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:25:14.988606    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:25:44.989626 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:25:44.989543    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:26:14.988942 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:26:14.988903    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:26:44.989234 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:26:44.989197    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:27:14.987947 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:27:14.987870    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:27:44.988123 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:27:44.988089    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:28:14.988023 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:28:14.987987    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:28:44.988104 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:28:44.988029    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:29:14.987963 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:29:14.987928    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:29:44.987728 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:29:44.987685    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:30:14.987992 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:30:14.987901    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:30:44.987750 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:30:44.987710    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:31:14.988756 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:31:14.988722    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:31:44.988383 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:31:44.988288    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:32:14.988241 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:32:14.988204    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:32:39.084015 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.083980    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"]
Nov 01 00:32:39.087207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.087191    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.140451 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.140417    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 00:32:39.190402 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.190364    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 00:32:39.190580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.190556    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-7zbpz\""
Nov 01 00:32:39.191352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.191332    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"]
Nov 01 00:32:39.250797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.250759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.251004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.250812    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4lp\" (UniqueName: \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.251004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.250918    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.351537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.351443    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4lp\" (UniqueName: \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.351537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.351501    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.351762 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.351542    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.351912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.351890    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.351976 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.351911    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.413044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.413010    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4lp\" (UniqueName: \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.665065 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.664989    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:32:39.668125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.668109    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.702746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.702712    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:32:39.722925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722900    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722940    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722901    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722970    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lqx6r\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722942    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722900    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 00:32:39.723059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.722941    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 00:32:39.739689 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.739651    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:32:39.856000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.855964    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2f6h\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856017    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856076    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856094    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856304 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856195    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856304 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856225    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.856304 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.856253    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957244    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957276    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957303    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957357    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957392    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957429    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2f6h\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957495    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.957973 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.957938    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.959964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.959934    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.960085 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.960033    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.960371 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.960355    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:39.960444 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:39.960407    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:40.018795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.018764    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:40.018947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.018830    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2f6h\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\") pod \"istiod-openshift-gateway-7876bfcb74-68mj5\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:40.075646 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.075619    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"]
Nov 01 00:32:40.078829 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:32:40.078789    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bcef4b_738a_4d34_aa8a_0d328e231909.slice/crio-8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4 WatchSource:0}: Error finding container 8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4: Status 404 returned error can't find the container with id 8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4
Nov 01 00:32:40.080691 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.080677    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:32:40.280092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.280062    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:40.307504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.307474    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerStarted","Data":"8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4"}
Nov 01 00:32:40.706407 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:40.706327    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:32:40.709169 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:32:40.709115    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5885ffa4_f7cf_402d_b0d2_1f4e7cb11e30.slice/crio-cef108dbc971fd058df0ee72eabf5bf9de9cd2d7d6e485263a2889c8bb0d940e WatchSource:0}: Error finding container cef108dbc971fd058df0ee72eabf5bf9de9cd2d7d6e485263a2889c8bb0d940e: Status 404 returned error can't find the container with id cef108dbc971fd058df0ee72eabf5bf9de9cd2d7d6e485263a2889c8bb0d940e
Nov 01 00:32:41.313413 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:41.313372    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" event={"ID":"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30","Type":"ContainerStarted","Data":"cef108dbc971fd058df0ee72eabf5bf9de9cd2d7d6e485263a2889c8bb0d940e"}
Nov 01 00:32:44.968515 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:44.968480    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:32:44.968888 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:44.968565    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:32:44.994436 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:32:44.994401    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:32:45.332343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:45.332299    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" event={"ID":"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30","Type":"ContainerStarted","Data":"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"}
Nov 01 00:32:45.332537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:45.332448    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:45.556283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:45.556216    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" podStartSLOduration=2.300292649 podStartE2EDuration="6.556192759s" totalImagesPullingTime="4.25590011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:32:45.505665384 +0000 UTC m=+555.183252329" watchObservedRunningTime="2026-11-01 00:32:45.556192759 +0000 UTC m=+555.233779707"
Nov 01 00:32:46.339144 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:46.339090    4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-68mj5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:32:46.339729 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:46.339173    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:32:47.342176 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:47.342122    4493 generic.go:356] "Generic (PLEG): container finished" podID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerID="32fed1893b0d67c372004cb063d9b5cf0f204e6be28941bdc0fcaeaafa68bdac" exitCode=0
Nov 01 00:32:47.342176 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:47.342166    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerDied","Data":"32fed1893b0d67c372004cb063d9b5cf0f204e6be28941bdc0fcaeaafa68bdac"}
Nov 01 00:32:49.338775 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:49.338737    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:32:51.357306 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:51.357272    4493 generic.go:356] "Generic (PLEG): container finished" podID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerID="7153b3613d51aba47cc498844b2cf5eac4fa33ab3ca161ce60bd15d627072a94" exitCode=0
Nov 01 00:32:51.357699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:51.357331    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerDied","Data":"7153b3613d51aba47cc498844b2cf5eac4fa33ab3ca161ce60bd15d627072a94"}
Nov 01 00:32:59.388081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:59.388044    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerStarted","Data":"f22769b28a04dbe5e905431a43f4f2b9625dbdb636485fba08dc6c589fdc27d8"}
Nov 01 00:32:59.507438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:32:59.507382    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" podStartSLOduration=2.901836325 podStartE2EDuration="20.507368783s" totalImagesPullingTime="17.605532458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:32:59.506640329 +0000 UTC m=+569.184227275" watchObservedRunningTime="2026-11-01 00:32:59.507368783 +0000 UTC m=+569.184955769"
Nov 01 00:33:00.393870 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:00.393835    4493 generic.go:356] "Generic (PLEG): container finished" podID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerID="f22769b28a04dbe5e905431a43f4f2b9625dbdb636485fba08dc6c589fdc27d8" exitCode=0
Nov 01 00:33:00.394289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:00.393919    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerDied","Data":"f22769b28a04dbe5e905431a43f4f2b9625dbdb636485fba08dc6c589fdc27d8"}
Nov 01 00:33:01.591705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.591667    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:33:01.615299 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.615261    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\" (UniqueName: \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\") pod \"17bcef4b-738a-4d34-aa8a-0d328e231909\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") "
Nov 01 00:33:01.615299 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.615303    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\") pod \"17bcef4b-738a-4d34-aa8a-0d328e231909\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") "
Nov 01 00:33:01.615502 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.615347    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\") pod \"17bcef4b-738a-4d34-aa8a-0d328e231909\" (UID: \"17bcef4b-738a-4d34-aa8a-0d328e231909\") "
Nov 01 00:33:01.615956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.615922    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle" pod "17bcef4b-738a-4d34-aa8a-0d328e231909" (UID: "17bcef4b-738a-4d34-aa8a-0d328e231909"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:33:01.617607 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.617581    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp" pod "17bcef4b-738a-4d34-aa8a-0d328e231909" (UID: "17bcef4b-738a-4d34-aa8a-0d328e231909"). InnerVolumeSpecName "kube-api-access-js4lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:33:01.620344 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.620319    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util" pod "17bcef4b-738a-4d34-aa8a-0d328e231909" (UID: "17bcef4b-738a-4d34-aa8a-0d328e231909"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:33:01.716118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.716028    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4lp\" (UniqueName: \"kubernetes.io/projected/17bcef4b-738a-4d34-aa8a-0d328e231909-kube-api-access-js4lp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:33:01.716118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.716056    4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-util\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:33:01.716118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:01.716066    4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17bcef4b-738a-4d34-aa8a-0d328e231909-bundle\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:33:02.401791 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:02.401763    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf"
Nov 01 00:33:02.401958 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:02.401763    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csppjf" event={"ID":"17bcef4b-738a-4d34-aa8a-0d328e231909","Type":"ContainerDied","Data":"8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4"}
Nov 01 00:33:02.401958 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:02.401895    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e439a38d92c7b44dcf006755631dc2572a04c6f9c10eefce14436d5150ed3f4"
Nov 01 00:33:14.988763 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:33:14.988663    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:33:34.925345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925306    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"]
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925555    4493 cpu_manager.go:414] "containerMap: removing container" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="util"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925565    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="util"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925573    4493 cpu_manager.go:414] "containerMap: removing container" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="pull"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925578    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="pull"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925598    4493 cpu_manager.go:414] "containerMap: removing container" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="extract"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925603    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="extract"
Nov 01 00:33:34.925761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.925644    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bcef4b-738a-4d34-aa8a-0d328e231909" containerName="extract"
Nov 01 00:33:34.927494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.927479    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:34.980483 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980448    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 00:33:34.980646 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980533    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 00:33:34.980700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980678    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 00:33:34.980789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980775    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 00:33:34.980837 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980791    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 00:33:34.980837 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:34.980828    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-28jl7\""
Nov 01 00:33:35.032570 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.032539    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"]
Nov 01 00:33:35.069404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069370    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.069404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069405    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.069605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069427    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2ts\" (UniqueName: \"kubernetes.io/projected/da6e216e-f24a-4c74-97c0-7de00ca15f30-kube-api-access-5h2ts\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.069605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069450    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.069605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069516    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.069732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.069612    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.169992    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170032    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170052    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2ts\" (UniqueName: \"kubernetes.io/projected/da6e216e-f24a-4c74-97c0-7de00ca15f30-kube-api-access-5h2ts\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170070    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170260    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.170635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.170606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.172733 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.172703    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6e216e-f24a-4c74-97c0-7de00ca15f30-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.172886 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.172858    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.173199 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.173182    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.173261 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.173243    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6e216e-f24a-4c74-97c0-7de00ca15f30-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.235041 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.235011    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2ts\" (UniqueName: \"kubernetes.io/projected/da6e216e-f24a-4c74-97c0-7de00ca15f30-kube-api-access-5h2ts\") pod \"opendatahub-operator-controller-manager-7b76c875b9-dpkl7\" (UID: \"da6e216e-f24a-4c74-97c0-7de00ca15f30\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.240876 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.240843    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:35.741479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:35.741438    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"]
Nov 01 00:33:35.744377 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:33:35.744354    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e216e_f24a_4c74_97c0_7de00ca15f30.slice/crio-da69fff22ccbe437a427f571cebbce1057a7ffe82c908f09cdbaeda4fb73bd51 WatchSource:0}: Error finding container da69fff22ccbe437a427f571cebbce1057a7ffe82c908f09cdbaeda4fb73bd51: Status 404 returned error can't find the container with id da69fff22ccbe437a427f571cebbce1057a7ffe82c908f09cdbaeda4fb73bd51
Nov 01 00:33:36.516880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:36.516837    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" event={"ID":"da6e216e-f24a-4c74-97c0-7de00ca15f30","Type":"ContainerStarted","Data":"da69fff22ccbe437a427f571cebbce1057a7ffe82c908f09cdbaeda4fb73bd51"}
Nov 01 00:33:39.529964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:39.529928    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" event={"ID":"da6e216e-f24a-4c74-97c0-7de00ca15f30","Type":"ContainerStarted","Data":"22fd4b7f19d0dc48a217728293c8937416e2e434d1d63f6731e8c2e7aa175a5c"}
Nov 01 00:33:40.534401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:40.534364    4493 generic.go:356] "Generic (PLEG): container finished" podID="da6e216e-f24a-4c74-97c0-7de00ca15f30" containerID="22fd4b7f19d0dc48a217728293c8937416e2e434d1d63f6731e8c2e7aa175a5c" exitCode=0
Nov 01 00:33:40.535081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:40.534436    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" event={"ID":"da6e216e-f24a-4c74-97c0-7de00ca15f30","Type":"ContainerDied","Data":"22fd4b7f19d0dc48a217728293c8937416e2e434d1d63f6731e8c2e7aa175a5c"}
Nov 01 00:33:41.540282 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:41.540245    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" event={"ID":"da6e216e-f24a-4c74-97c0-7de00ca15f30","Type":"ContainerStarted","Data":"a72d0eabd2cd1ee3df150ed733c91133a62516b8026cfba214e98ea4323bd4e7"}
Nov 01 00:33:41.540699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:41.540376    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:33:41.789423 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:41.789372    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" podStartSLOduration=2.75705727 podStartE2EDuration="7.789356816s" totalImagesPullingTime="4.523585291s" totalInitContainerRuntime="508.714255ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:41.739583165 +0000 UTC m=+611.417170109" watchObservedRunningTime="2026-11-01 00:33:41.789356816 +0000 UTC m=+611.466943761"
Nov 01 00:33:44.988749 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:33:44.988713    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:33:52.546778 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:33:52.546749    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7"
Nov 01 00:34:13.765513 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.765479    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"]
Nov 01 00:34:13.771413 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.771394    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.859311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.859279    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"]
Nov 01 00:34:13.861585 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.861556    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29781e95-2467-4abd-ba34-e73bf9812f2f-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.861732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.861642    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29781e95-2467-4abd-ba34-e73bf9812f2f-tmp\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.861732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.861698    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mt9r\" (UniqueName: \"kubernetes.io/projected/29781e95-2467-4abd-ba34-e73bf9812f2f-kube-api-access-4mt9r\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.911293 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.911266    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 00:34:13.911461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.911322    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 00:34:13.911461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.911330    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9sfnp\""
Nov 01 00:34:13.962282 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.962248    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mt9r\" (UniqueName: \"kubernetes.io/projected/29781e95-2467-4abd-ba34-e73bf9812f2f-kube-api-access-4mt9r\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.962474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.962331    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29781e95-2467-4abd-ba34-e73bf9812f2f-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.962474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.962361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29781e95-2467-4abd-ba34-e73bf9812f2f-tmp\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.964872 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.964846    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29781e95-2467-4abd-ba34-e73bf9812f2f-tmp\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:13.965012 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:13.964941    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29781e95-2467-4abd-ba34-e73bf9812f2f-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:14.023000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:14.022924    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mt9r\" (UniqueName: \"kubernetes.io/projected/29781e95-2467-4abd-ba34-e73bf9812f2f-kube-api-access-4mt9r\") pod \"kube-auth-proxy-69c59cdf59-jk54h\" (UID: \"29781e95-2467-4abd-ba34-e73bf9812f2f\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:14.081912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:14.081877    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"
Nov 01 00:34:14.452856 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:14.452774    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h"]
Nov 01 00:34:14.455864 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:34:14.455838    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29781e95_2467_4abd_ba34_e73bf9812f2f.slice/crio-37c19340ce7f3e5255e08aa4c7552c48e6af08fb56ba460ea4fd26bdebfae55b WatchSource:0}: Error finding container 37c19340ce7f3e5255e08aa4c7552c48e6af08fb56ba460ea4fd26bdebfae55b: Status 404 returned error can't find the container with id 37c19340ce7f3e5255e08aa4c7552c48e6af08fb56ba460ea4fd26bdebfae55b
Nov 01 00:34:14.652052 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:14.652010    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h" event={"ID":"29781e95-2467-4abd-ba34-e73bf9812f2f","Type":"ContainerStarted","Data":"37c19340ce7f3e5255e08aa4c7552c48e6af08fb56ba460ea4fd26bdebfae55b"}
Nov 01 00:34:14.989103 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:34:14.989069    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:34:17.665067 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:17.665032    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h" event={"ID":"29781e95-2467-4abd-ba34-e73bf9812f2f","Type":"ContainerStarted","Data":"2ef45babaa630c6e9942bb41e4829264a31d2cdab869f8e250424ad877bd2f6b"}
Nov 01 00:34:17.781038 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:17.780977    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-jk54h" podStartSLOduration=1.813198426 podStartE2EDuration="4.780958813s" totalImagesPullingTime="2.967760387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:17.780405291 +0000 UTC m=+647.457992250" watchObservedRunningTime="2026-11-01 00:34:17.780958813 +0000 UTC m=+647.458545759"
Nov 01 00:34:36.380791 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.380757    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:36.385379 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.385362    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:36.437774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.437731    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 00:34:36.437774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.437731    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 00:34:36.437988 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.437903    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2d4\" (UniqueName: \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\") pod \"authorino-57f7f8659f-qwxk6\" (UID: \"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac\") " pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:36.438818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.438796    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:36.487605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.487573    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qwcjk\""
Nov 01 00:34:36.539213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.539176    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2d4\" (UniqueName: \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\") pod \"authorino-57f7f8659f-qwxk6\" (UID: \"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac\") " pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:36.600465 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.600431    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2d4\" (UniqueName: \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\") pod \"authorino-57f7f8659f-qwxk6\" (UID: \"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac\") " pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:36.716383 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:36.716282    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:37.085849 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:37.085820    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:37.088988 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:34:37.088961    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ca2ba3_4a41_41d5_809b_62373ae2d9ac.slice/crio-6d0a1dbaa66a50c1903339b3e59ea8c4c1785882f3d0eb5269252cc06f71ea00 WatchSource:0}: Error finding container 6d0a1dbaa66a50c1903339b3e59ea8c4c1785882f3d0eb5269252cc06f71ea00: Status 404 returned error can't find the container with id 6d0a1dbaa66a50c1903339b3e59ea8c4c1785882f3d0eb5269252cc06f71ea00
Nov 01 00:34:37.739877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:37.739838    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" event={"ID":"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac","Type":"ContainerStarted","Data":"6d0a1dbaa66a50c1903339b3e59ea8c4c1785882f3d0eb5269252cc06f71ea00"}
Nov 01 00:34:38.037795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.037722    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:34:38.042045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.042021    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:34:38.042181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.042154    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:34:38.044826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.044807    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.097009 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.096976    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 00:34:38.097961 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.097939    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:34:38.146761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.146725    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gp6r7\""
Nov 01 00:34:38.147945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.147924    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:34:38.153031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.153012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8cc39922-b719-4e48-8907-ee00f259cb9c-config-file\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.153101 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.153057    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6gj\" (UniqueName: \"kubernetes.io/projected/8cc39922-b719-4e48-8907-ee00f259cb9c-kube-api-access-nt6gj\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.153101 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.153095    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zns\" (UniqueName: \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\") pod \"authorino-56c6c9ddd5-plz8x\" (UID: \"06d59e98-4a79-4acb-8743-146dd13edd4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:34:38.253910 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.253873    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6gj\" (UniqueName: \"kubernetes.io/projected/8cc39922-b719-4e48-8907-ee00f259cb9c-kube-api-access-nt6gj\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.254086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.253921    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zns\" (UniqueName: \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\") pod \"authorino-56c6c9ddd5-plz8x\" (UID: \"06d59e98-4a79-4acb-8743-146dd13edd4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:34:38.254086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.253958    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8cc39922-b719-4e48-8907-ee00f259cb9c-config-file\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.254599 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.254581    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8cc39922-b719-4e48-8907-ee00f259cb9c-config-file\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.314967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.314896    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zns\" (UniqueName: \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\") pod \"authorino-56c6c9ddd5-plz8x\" (UID: \"06d59e98-4a79-4acb-8743-146dd13edd4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:34:38.315098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.315055    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6gj\" (UniqueName: \"kubernetes.io/projected/8cc39922-b719-4e48-8907-ee00f259cb9c-kube-api-access-nt6gj\") pod \"limitador-limitador-7c9b87d574-v4cj6\" (UID: \"8cc39922-b719-4e48-8907-ee00f259cb9c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.354892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.354852    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:34:38.361014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.360992    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:38.364443 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.364274    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:34:38.728666 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.728636    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:34:38.731424 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:34:38.731395    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06d59e98_4a79_4acb_8743_146dd13edd4c.slice/crio-4c964d54429c16d1e71281fe9e4e71820415af2d6331a53548b36ad0a9a9989e WatchSource:0}: Error finding container 4c964d54429c16d1e71281fe9e4e71820415af2d6331a53548b36ad0a9a9989e: Status 404 returned error can't find the container with id 4c964d54429c16d1e71281fe9e4e71820415af2d6331a53548b36ad0a9a9989e
Nov 01 00:34:38.744405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.744375    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" event={"ID":"06d59e98-4a79-4acb-8743-146dd13edd4c","Type":"ContainerStarted","Data":"4c964d54429c16d1e71281fe9e4e71820415af2d6331a53548b36ad0a9a9989e"}
Nov 01 00:34:38.780427 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:38.780402    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:34:38.783518 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:34:38.783491    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc39922_b719_4e48_8907_ee00f259cb9c.slice/crio-b21d055bb73ecc7910427c312b22258ba290abf3c0944bc39fa4a416fda0028e WatchSource:0}: Error finding container b21d055bb73ecc7910427c312b22258ba290abf3c0944bc39fa4a416fda0028e: Status 404 returned error can't find the container with id b21d055bb73ecc7910427c312b22258ba290abf3c0944bc39fa4a416fda0028e
Nov 01 00:34:39.750724 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:39.750687    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6" event={"ID":"8cc39922-b719-4e48-8907-ee00f259cb9c","Type":"ContainerStarted","Data":"b21d055bb73ecc7910427c312b22258ba290abf3c0944bc39fa4a416fda0028e"}
Nov 01 00:34:41.761270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:41.761187    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6" event={"ID":"8cc39922-b719-4e48-8907-ee00f259cb9c","Type":"ContainerStarted","Data":"e1aa0d74b15c00ad5bbf93fad4814a0cc8a4c55e36499191da70104029745a8a"}
Nov 01 00:34:41.761671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:41.761290    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:34:41.878581 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:41.878528    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6" podStartSLOduration=2.009701068 podStartE2EDuration="4.87851212s" totalImagesPullingTime="2.868811052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:41.877457851 +0000 UTC m=+671.555044798" watchObservedRunningTime="2026-11-01 00:34:41.87851212 +0000 UTC m=+671.556099100"
Nov 01 00:34:43.771031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:43.770997    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" event={"ID":"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac","Type":"ContainerStarted","Data":"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"}
Nov 01 00:34:43.772600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:43.772578    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" event={"ID":"06d59e98-4a79-4acb-8743-146dd13edd4c","Type":"ContainerStarted","Data":"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"}
Nov 01 00:34:43.885625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:43.885569    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" podStartSLOduration=1.6409665009999999 podStartE2EDuration="7.885555954s" totalImagesPullingTime="6.244589453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:43.884530126 +0000 UTC m=+673.562117066" watchObservedRunningTime="2026-11-01 00:34:43.885555954 +0000 UTC m=+673.563142898"
Nov 01 00:34:43.999612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:43.999546    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" podStartSLOduration=2.405542702 podStartE2EDuration="6.999526913s" totalImagesPullingTime="4.593984211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:43.998171067 +0000 UTC m=+673.675758012" watchObservedRunningTime="2026-11-01 00:34:43.999526913 +0000 UTC m=+673.677113862"
Nov 01 00:34:44.022181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:44.022060    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:44.987900 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:34:44.987867    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:34:45.780067 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:45.780030    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" podUID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" containerName="authorino" containerID="cri-o://dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276" gracePeriod=30
Nov 01 00:34:46.017719 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.017693    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:46.120312 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.120218    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\" (UniqueName: \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\") pod \"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac\" (UID: \"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac\") "
Nov 01 00:34:46.122578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.122550    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4" pod "f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" (UID: "f0ca2ba3-4a41-41d5-809b-62373ae2d9ac"). InnerVolumeSpecName "kube-api-access-vp2d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:34:46.221237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.221194    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp2d4\" (UniqueName: \"kubernetes.io/projected/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac-kube-api-access-vp2d4\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:34:46.784732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.784633    4493 generic.go:356] "Generic (PLEG): container finished" podID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" containerID="dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276" exitCode=0
Nov 01 00:34:46.784732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.784678    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwxk6"
Nov 01 00:34:46.784732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.784714    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" event={"ID":"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac","Type":"ContainerDied","Data":"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"}
Nov 01 00:34:46.784990 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.784758    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwxk6" event={"ID":"f0ca2ba3-4a41-41d5-809b-62373ae2d9ac","Type":"ContainerDied","Data":"6d0a1dbaa66a50c1903339b3e59ea8c4c1785882f3d0eb5269252cc06f71ea00"}
Nov 01 00:34:46.784990 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.784775    4493 scope.go:122] "RemoveContainer" containerID="dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"
Nov 01 00:34:46.792806 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.792789    4493 scope.go:122] "RemoveContainer" containerID="dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"
Nov 01 00:34:46.793075 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:34:46.793057    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276\": container with ID starting with dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276 not found: ID does not exist" containerID="dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"
Nov 01 00:34:46.793124 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.793089    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276"} err="failed to get container status \"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276\": rpc error: code = NotFound desc = could not find container \"dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276\": container with ID starting with dbdd28eb210beb8c31ac32d1df57458f1a1750d45c04914d56ea49b38243a276 not found: ID does not exist"
Nov 01 00:34:46.947665 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.947637    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:46.951613 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:46.951587    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwxk6"]
Nov 01 00:34:48.500849 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:48.500813    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" path="/var/lib/kubelet/pods/f0ca2ba3-4a41-41d5-809b-62373ae2d9ac/volumes"
Nov 01 00:34:52.767109 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:34:52.767071    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"
Nov 01 00:35:14.989744 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:35:14.989705    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:35:43.763514 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.763470    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"]
Nov 01 00:35:43.764016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.763862    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" containerName="authorino"
Nov 01 00:35:43.764016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.763879    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" containerName="authorino"
Nov 01 00:35:43.764016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.763975    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ca2ba3-4a41-41d5-809b-62373ae2d9ac" containerName="authorino"
Nov 01 00:35:43.768488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.768464    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.872426 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.872392    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"]
Nov 01 00:35:43.878320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878294    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878335    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/044a721d-67d4-4b5d-98ee-49efb2d0d746-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878438    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878592    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878635    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.878705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.878660    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mp7\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-kube-api-access-z2mp7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979416 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979379    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979416 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/044a721d-67d4-4b5d-98ee-49efb2d0d746-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979545    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979589    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979617    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mp7\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-kube-api-access-z2mp7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.979858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.979697    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.980348 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.980324    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.981849 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.981827    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/044a721d-67d4-4b5d-98ee-49efb2d0d746-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.982033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.982015    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.982460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.982445    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:43.982579 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:43.982560    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/044a721d-67d4-4b5d-98ee-49efb2d0d746-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:44.040539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.040465    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:44.040738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.040718    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mp7\" (UniqueName: \"kubernetes.io/projected/044a721d-67d4-4b5d-98ee-49efb2d0d746-kube-api-access-z2mp7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x62wb\" (UID: \"044a721d-67d4-4b5d-98ee-49efb2d0d746\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:44.080792 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.080761    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:44.503291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.503260    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"]
Nov 01 00:35:44.506024 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:35:44.505998    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044a721d_67d4_4b5d_98ee_49efb2d0d746.slice/crio-934a8e2f429378cd7224f4fe9b936a79ac19ee71a25fb6ef43dc68f3cc18d01c WatchSource:0}: Error finding container 934a8e2f429378cd7224f4fe9b936a79ac19ee71a25fb6ef43dc68f3cc18d01c: Status 404 returned error can't find the container with id 934a8e2f429378cd7224f4fe9b936a79ac19ee71a25fb6ef43dc68f3cc18d01c
Nov 01 00:35:44.508085 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.508055    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:35:44.508198 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:44.508119    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:35:44.988453 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:35:44.988421    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:35:45.017256 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.017227    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb" event={"ID":"044a721d-67d4-4b5d-98ee-49efb2d0d746","Type":"ContainerStarted","Data":"c76efb767b28bc86413a4e6494d2f8f95222e1e3074d5e8140c4f0c58e36ba0c"}
Nov 01 00:35:45.017448 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.017278    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb" event={"ID":"044a721d-67d4-4b5d-98ee-49efb2d0d746","Type":"ContainerStarted","Data":"934a8e2f429378cd7224f4fe9b936a79ac19ee71a25fb6ef43dc68f3cc18d01c"}
Nov 01 00:35:45.017448 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.017426    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:45.018976 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.018955    4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-x62wb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:35:45.019092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.019000    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb" podUID="044a721d-67d4-4b5d-98ee-49efb2d0d746" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:35:45.144525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:45.144469    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb" podStartSLOduration=2.144451547 podStartE2EDuration="2.144451547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:45.143947163 +0000 UTC m=+734.821534108" watchObservedRunningTime="2026-11-01 00:35:45.144451547 +0000 UTC m=+734.822038493"
Nov 01 00:35:46.021882 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.021857    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x62wb"
Nov 01 00:35:46.244383 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.244355    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:35:46.244550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.244532    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery" containerID="cri-o://9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1" gracePeriod=30
Nov 01 00:35:46.337527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.337499    4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-68mj5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused" start-of-body=
Nov 01 00:35:46.337641 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.337552    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused"
Nov 01 00:35:46.478252 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.478228    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:35:46.600312 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600226    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600312 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600280    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600316    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600380    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600407    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600463    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600498    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\") pod \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\" (UID: \"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30\") "
Nov 01 00:35:46.600829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600655    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:35:46.600829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.600806    4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-ca-configmap\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.604232 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.604178    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:35:46.604380 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.604233    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:35:46.604653 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.604619    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "kube-api-access-v2f6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:35:46.604858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.604810    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:35:46.605033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.605008    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:35:46.606092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.606065    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig" pod "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" (UID: "5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:35:46.701604 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701573    4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-csr-dns-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.701604 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701601    4493 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-local-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.701604 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701611    4493 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-kubeconfig\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.701824 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701619    4493 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-cacerts\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.701824 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701627    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:46.701824 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:46.701635    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2f6h\" (UniqueName: \"kubernetes.io/projected/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30-kube-api-access-v2f6h\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:35:47.026058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.026026    4493 generic.go:356] "Generic (PLEG): container finished" podID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerID="9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1" exitCode=0
Nov 01 00:35:47.026490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.026064    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"
Nov 01 00:35:47.026490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.026105    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" event={"ID":"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30","Type":"ContainerDied","Data":"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"}
Nov 01 00:35:47.026490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.026168    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5" event={"ID":"5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30","Type":"ContainerDied","Data":"cef108dbc971fd058df0ee72eabf5bf9de9cd2d7d6e485263a2889c8bb0d940e"}
Nov 01 00:35:47.026490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.026190    4493 scope.go:122] "RemoveContainer" containerID="9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"
Nov 01 00:35:47.035053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.035036    4493 scope.go:122] "RemoveContainer" containerID="9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"
Nov 01 00:35:47.035343 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:35:47.035328    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1\": container with ID starting with 9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1 not found: ID does not exist" containerID="9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"
Nov 01 00:35:47.035398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.035352    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1"} err="failed to get container status \"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1\": rpc error: code = NotFound desc = could not find container \"9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1\": container with ID starting with 9e974b289b9ba6669c0440fc5aa35ac49d9599b1580b825fa676f92b6eb53ae1 not found: ID does not exist"
Nov 01 00:35:47.190554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.190518    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:35:47.240849 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:47.240820    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68mj5"]
Nov 01 00:35:48.503344 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:48.501032    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" path="/var/lib/kubelet/pods/5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30/volumes"
Nov 01 00:35:49.787787 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.787750    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-qxkrv"]
Nov 01 00:35:49.788202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.788042    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery"
Nov 01 00:35:49.788202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.788056    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery"
Nov 01 00:35:49.788202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.788146    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5885ffa4-f7cf-402d-b0d2-1f4e7cb11e30" containerName="discovery"
Nov 01 00:35:49.792145 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.792117    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:49.846326 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.846255    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 00:35:49.896291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.896255    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 00:35:49.896469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.896317    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dcgvz\""
Nov 01 00:35:49.896469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.896318    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 00:35:49.896469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.896359    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 00:35:49.897525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.897510    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qxkrv"]
Nov 01 00:35:49.925788 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.925763    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwb2\" (UniqueName: \"kubernetes.io/projected/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-kube-api-access-ttwb2\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:49.925868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.925790    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-cert\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:49.925868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:49.925820    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-metrics-certs\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.026583 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.026545    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-cert\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.026748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.026597    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-metrics-certs\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.026748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.026653    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwb2\" (UniqueName: \"kubernetes.io/projected/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-kube-api-access-ttwb2\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.029308 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.029281    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-cert\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.029406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.029281    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-metrics-certs\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.087934 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.087865    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwb2\" (UniqueName: \"kubernetes.io/projected/fdb7b4c5-a144-4029-a2f2-46895f82ef7a-kube-api-access-ttwb2\") pod \"odh-model-controller-769875db56-qxkrv\" (UID: \"fdb7b4c5-a144-4029-a2f2-46895f82ef7a\") " pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.104730 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.104699    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:50.476024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:50.475999    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qxkrv"]
Nov 01 00:35:50.479057 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:35:50.479035    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb7b4c5_a144_4029_a2f2_46895f82ef7a.slice/crio-b1b3ef8754774cf9ba51e74c9f7ca4b9d64f6a569d2aa0b1e736ddfefecb83c1 WatchSource:0}: Error finding container b1b3ef8754774cf9ba51e74c9f7ca4b9d64f6a569d2aa0b1e736ddfefecb83c1: Status 404 returned error can't find the container with id b1b3ef8754774cf9ba51e74c9f7ca4b9d64f6a569d2aa0b1e736ddfefecb83c1
Nov 01 00:35:51.043042 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:51.043007    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qxkrv" event={"ID":"fdb7b4c5-a144-4029-a2f2-46895f82ef7a","Type":"ContainerStarted","Data":"b1b3ef8754774cf9ba51e74c9f7ca4b9d64f6a569d2aa0b1e736ddfefecb83c1"}
Nov 01 00:35:54.058740 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:54.058704    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qxkrv" event={"ID":"fdb7b4c5-a144-4029-a2f2-46895f82ef7a","Type":"ContainerStarted","Data":"b5a6b93043373b39282597ef5c3df21847cd4042cd1f57b2af2f97e021110b03"}
Nov 01 00:35:54.059125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:54.058863    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:35:54.176088 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:35:54.176033    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-qxkrv" podStartSLOduration=2.172837281 podStartE2EDuration="5.176017023s" totalImagesPullingTime="3.003179742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:54.174655334 +0000 UTC m=+743.852242279" watchObservedRunningTime="2026-11-01 00:35:54.176017023 +0000 UTC m=+743.853603965"
Nov 01 00:36:04.056764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.056729    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:04.060118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.060097    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.153987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.153951    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.154188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.154010    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.154188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.154028    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.154188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.154104    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tc5\" (UniqueName: \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.170105 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.170068    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 00:36:04.170105 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.170103    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 00:36:04.170333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.170171    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 00:36:04.170333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.170267    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jq2v\""
Nov 01 00:36:04.170333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.170300    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 00:36:04.171482 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.171464    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:04.255342 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.255305    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.255509 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.255395    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.255509 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.255426    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.255509 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.255452    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tc5\" (UniqueName: \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.256157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.256094    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.257896 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.257871    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.258021 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.257915    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.317043 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.316967    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tc5\" (UniqueName: \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\") pod \"maas-api-6d669c7fcd-dq6j7\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.372907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.372871    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:04.745613 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:04.745582    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:04.748940 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:36:04.748904    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7963a061_dbb8_4e6a_a914_83a22de62335.slice/crio-981e232337b0caa0494934b42108183216de3bc5942e1ada01a9537d08413f80 WatchSource:0}: Error finding container 981e232337b0caa0494934b42108183216de3bc5942e1ada01a9537d08413f80: Status 404 returned error can't find the container with id 981e232337b0caa0494934b42108183216de3bc5942e1ada01a9537d08413f80
Nov 01 00:36:05.064493 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:05.064414    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-qxkrv"
Nov 01 00:36:05.100464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:05.100432    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" event={"ID":"7963a061-dbb8-4e6a-a914-83a22de62335","Type":"ContainerStarted","Data":"981e232337b0caa0494934b42108183216de3bc5942e1ada01a9537d08413f80"}
Nov 01 00:36:09.120031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:09.119997    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" event={"ID":"7963a061-dbb8-4e6a-a914-83a22de62335","Type":"ContainerStarted","Data":"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"}
Nov 01 00:36:09.120481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:09.120100    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:09.237923 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:09.237858    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" podStartSLOduration=1.322315822 podStartE2EDuration="5.237838686s" totalImagesPullingTime="3.915522864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:36:09.236874632 +0000 UTC m=+758.914461577" watchObservedRunningTime="2026-11-01 00:36:09.237838686 +0000 UTC m=+758.915425635"
Nov 01 00:36:09.561563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:09.561527    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:36:14.989450 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:14.989370    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:36:15.129937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:15.129907    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:15.938454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:15.938421    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:15.941657 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:15.941636    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:15.997004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:15.996969    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:16.050391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.050352    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5l58\" (UniqueName: \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\") pod \"authorino-6f6944bcdd-slmzw\" (UID: \"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d\") " pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:16.151330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.151285    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5l58\" (UniqueName: \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\") pod \"authorino-6f6944bcdd-slmzw\" (UID: \"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d\") " pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:16.212934 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.212859    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5l58\" (UniqueName: \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\") pod \"authorino-6f6944bcdd-slmzw\" (UID: \"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d\") " pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:16.252457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.252423    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:16.308265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.305643    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:16.330311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.330283    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:16.334090 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.334071    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:16.390781 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.390751    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:16.453708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.453668    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4nw\" (UniqueName: \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\") pod \"authorino-5ff4f9f84d-j977b\" (UID: \"70706d89-ccba-4014-b883-6fd5ff43b97a\") " pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:16.554769 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.554732    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4nw\" (UniqueName: \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\") pod \"authorino-5ff4f9f84d-j977b\" (UID: \"70706d89-ccba-4014-b883-6fd5ff43b97a\") " pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:16.616411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.616372    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4nw\" (UniqueName: \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\") pod \"authorino-5ff4f9f84d-j977b\" (UID: \"70706d89-ccba-4014-b883-6fd5ff43b97a\") " pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:16.621941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.621911    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:16.625599 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:36:16.625573    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7639bfe_32aa_4cb7_b555_307a9bf0aa2d.slice/crio-8e357be4e6221fd7e0b9b402a17b5899d22862ce65d16ceaed161904f62a7425 WatchSource:0}: Error finding container 8e357be4e6221fd7e0b9b402a17b5899d22862ce65d16ceaed161904f62a7425: Status 404 returned error can't find the container with id 8e357be4e6221fd7e0b9b402a17b5899d22862ce65d16ceaed161904f62a7425
Nov 01 00:36:16.645345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.645314    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:16.784880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.784847    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:16.812503 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.812429    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:36:16.815880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.815862    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:16.868365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.868340    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 00:36:16.872863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.872831    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:36:16.958020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.957987    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:16.958248 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:16.958043    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcf2z\" (UniqueName: \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.013025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.012991    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:17.015836 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:36:17.015808    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70706d89_ccba_4014_b883_6fd5ff43b97a.slice/crio-547536965a289e42f3b90da9500109b57852d19916b493778ec75f97eac6f8b9 WatchSource:0}: Error finding container 547536965a289e42f3b90da9500109b57852d19916b493778ec75f97eac6f8b9: Status 404 returned error can't find the container with id 547536965a289e42f3b90da9500109b57852d19916b493778ec75f97eac6f8b9
Nov 01 00:36:17.059004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.058971    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcf2z\" (UniqueName: \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.059190 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.059037    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.061608 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.061589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.120808 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.120731    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcf2z\" (UniqueName: \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\") pod \"authorino-5b99d64678-t8mjf\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") " pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.127554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.127534    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:36:17.159854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.159815    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" event={"ID":"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d","Type":"ContainerStarted","Data":"8e357be4e6221fd7e0b9b402a17b5899d22862ce65d16ceaed161904f62a7425"}
Nov 01 00:36:17.160997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.160970    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff4f9f84d-j977b" event={"ID":"70706d89-ccba-4014-b883-6fd5ff43b97a","Type":"ContainerStarted","Data":"547536965a289e42f3b90da9500109b57852d19916b493778ec75f97eac6f8b9"}
Nov 01 00:36:17.501040 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:17.500692    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:36:17.503632 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:36:17.503605    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod945d3aca_dad1_4177_9190_1e07888c42c6.slice/crio-051cf12be99dfd9ca32d1d8bdff02f8d2708d6a94db3918667799e01f9de63bb WatchSource:0}: Error finding container 051cf12be99dfd9ca32d1d8bdff02f8d2708d6a94db3918667799e01f9de63bb: Status 404 returned error can't find the container with id 051cf12be99dfd9ca32d1d8bdff02f8d2708d6a94db3918667799e01f9de63bb
Nov 01 00:36:18.165922 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:18.165889    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b99d64678-t8mjf" event={"ID":"945d3aca-dad1-4177-9190-1e07888c42c6","Type":"ContainerStarted","Data":"051cf12be99dfd9ca32d1d8bdff02f8d2708d6a94db3918667799e01f9de63bb"}
Nov 01 00:36:19.170831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.170791    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b99d64678-t8mjf" event={"ID":"945d3aca-dad1-4177-9190-1e07888c42c6","Type":"ContainerStarted","Data":"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"}
Nov 01 00:36:19.285095 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.285032    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b99d64678-t8mjf" podStartSLOduration=1.736914436 podStartE2EDuration="3.285013961s" totalImagesPullingTime="1.548099525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:36:19.284988134 +0000 UTC m=+768.962575067" watchObservedRunningTime="2026-11-01 00:36:19.285013961 +0000 UTC m=+768.962600909"
Nov 01 00:36:19.311694 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.311620    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:36:19.311836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.311802    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" podUID="06d59e98-4a79-4acb-8743-146dd13edd4c" containerName="authorino" containerID="cri-o://84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d" gracePeriod=30
Nov 01 00:36:19.554002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.553980    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:36:19.681499 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.681423    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\" (UniqueName: \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\") pod \"06d59e98-4a79-4acb-8743-146dd13edd4c\" (UID: \"06d59e98-4a79-4acb-8743-146dd13edd4c\") "
Nov 01 00:36:19.683594 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.683568    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns" pod "06d59e98-4a79-4acb-8743-146dd13edd4c" (UID: "06d59e98-4a79-4acb-8743-146dd13edd4c"). InnerVolumeSpecName "kube-api-access-h4zns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:36:19.782113 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:19.782064    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zns\" (UniqueName: \"kubernetes.io/projected/06d59e98-4a79-4acb-8743-146dd13edd4c-kube-api-access-h4zns\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:20.176191 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.176152    4493 generic.go:356] "Generic (PLEG): container finished" podID="06d59e98-4a79-4acb-8743-146dd13edd4c" containerID="84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d" exitCode=0
Nov 01 00:36:20.176671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.176204    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x"
Nov 01 00:36:20.176671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.176239    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" event={"ID":"06d59e98-4a79-4acb-8743-146dd13edd4c","Type":"ContainerDied","Data":"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"}
Nov 01 00:36:20.176671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.176298    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-plz8x" event={"ID":"06d59e98-4a79-4acb-8743-146dd13edd4c","Type":"ContainerDied","Data":"4c964d54429c16d1e71281fe9e4e71820415af2d6331a53548b36ad0a9a9989e"}
Nov 01 00:36:20.176671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.176321    4493 scope.go:122] "RemoveContainer" containerID="84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"
Nov 01 00:36:20.185386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.185364    4493 scope.go:122] "RemoveContainer" containerID="84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"
Nov 01 00:36:20.185721 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:20.185700    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d\": container with ID starting with 84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d not found: ID does not exist" containerID="84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"
Nov 01 00:36:20.185789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.185734    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d"} err="failed to get container status \"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d\": rpc error: code = NotFound desc = could not find container \"84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d\": container with ID starting with 84d22ee9a8620087640af66872e1c9eef0ede532ca80ae3aeef71b55894cc32d not found: ID does not exist"
Nov 01 00:36:20.338920 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.338837    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:36:20.346912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.346887    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-plz8x"]
Nov 01 00:36:20.501160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:20.501106    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d59e98-4a79-4acb-8743-146dd13edd4c" path="/var/lib/kubelet/pods/06d59e98-4a79-4acb-8743-146dd13edd4c/volumes"
Nov 01 00:36:26.401456 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.401420    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:26.401905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.401680    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" podUID="7963a061-dbb8-4e6a-a914-83a22de62335" containerName="maas-api" containerID="cri-o://cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f" gracePeriod=30
Nov 01 00:36:26.651105 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.651077    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:26.733883 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.733855    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\") pod \"7963a061-dbb8-4e6a-a914-83a22de62335\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") "
Nov 01 00:36:26.734039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.733917    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\") pod \"7963a061-dbb8-4e6a-a914-83a22de62335\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") "
Nov 01 00:36:26.734039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.733961    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\" (UniqueName: \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\") pod \"7963a061-dbb8-4e6a-a914-83a22de62335\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") "
Nov 01 00:36:26.734039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.734013    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\") pod \"7963a061-dbb8-4e6a-a914-83a22de62335\" (UID: \"7963a061-dbb8-4e6a-a914-83a22de62335\") "
Nov 01 00:36:26.734368 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.734340    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca" pod "7963a061-dbb8-4e6a-a914-83a22de62335" (UID: "7963a061-dbb8-4e6a-a914-83a22de62335"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:36:26.736501 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.736468    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5" pod "7963a061-dbb8-4e6a-a914-83a22de62335" (UID: "7963a061-dbb8-4e6a-a914-83a22de62335"). InnerVolumeSpecName "kube-api-access-54tc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:36:26.736594 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.736502    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls" pod "7963a061-dbb8-4e6a-a914-83a22de62335" (UID: "7963a061-dbb8-4e6a-a914-83a22de62335"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:36:26.736594 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.736521    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs" pod "7963a061-dbb8-4e6a-a914-83a22de62335" (UID: "7963a061-dbb8-4e6a-a914-83a22de62335"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:36:26.834935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.834893    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tc5\" (UniqueName: \"kubernetes.io/projected/7963a061-dbb8-4e6a-a914-83a22de62335-kube-api-access-54tc5\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:26.834935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.834926    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:26.834935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.834936    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7963a061-dbb8-4e6a-a914-83a22de62335-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:26.834935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:26.834945    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7963a061-dbb8-4e6a-a914-83a22de62335-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:27.206343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.206261    4493 generic.go:356] "Generic (PLEG): container finished" podID="7963a061-dbb8-4e6a-a914-83a22de62335" containerID="cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f" exitCode=0
Nov 01 00:36:27.206343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.206310    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"
Nov 01 00:36:27.206592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.206348    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" event={"ID":"7963a061-dbb8-4e6a-a914-83a22de62335","Type":"ContainerDied","Data":"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"}
Nov 01 00:36:27.206592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.206397    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7" event={"ID":"7963a061-dbb8-4e6a-a914-83a22de62335","Type":"ContainerDied","Data":"981e232337b0caa0494934b42108183216de3bc5942e1ada01a9537d08413f80"}
Nov 01 00:36:27.206592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.206419    4493 scope.go:122] "RemoveContainer" containerID="cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"
Nov 01 00:36:27.215471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.215445    4493 scope.go:122] "RemoveContainer" containerID="cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"
Nov 01 00:36:27.215916 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:27.215894    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f\": container with ID starting with cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f not found: ID does not exist" containerID="cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"
Nov 01 00:36:27.215989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.215926    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f"} err="failed to get container status \"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f\": rpc error: code = NotFound desc = could not find container \"cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f\": container with ID starting with cae7b0853e08ac10c1580a631c612c799106fbc1997187142a1e0b106b7a059f not found: ID does not exist"
Nov 01 00:36:27.370184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.370148    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:27.419938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:27.419908    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-dq6j7"]
Nov 01 00:36:28.501517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:28.501489    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7963a061-dbb8-4e6a-a914-83a22de62335" path="/var/lib/kubelet/pods/7963a061-dbb8-4e6a-a914-83a22de62335/volumes"
Nov 01 00:36:29.226920 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.226288    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff4f9f84d-j977b" event={"ID":"70706d89-ccba-4014-b883-6fd5ff43b97a","Type":"ContainerStarted","Data":"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"}
Nov 01 00:36:29.226920 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.226464    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5ff4f9f84d-j977b" podUID="70706d89-ccba-4014-b883-6fd5ff43b97a" containerName="authorino" containerID="cri-o://3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0" gracePeriod=30
Nov 01 00:36:29.482829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.482767    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:29.561795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.561766    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\" (UniqueName: \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\") pod \"70706d89-ccba-4014-b883-6fd5ff43b97a\" (UID: \"70706d89-ccba-4014-b883-6fd5ff43b97a\") "
Nov 01 00:36:29.564118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.564090    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw" pod "70706d89-ccba-4014-b883-6fd5ff43b97a" (UID: "70706d89-ccba-4014-b883-6fd5ff43b97a"). InnerVolumeSpecName "kube-api-access-pk4nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:36:29.662718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:29.662678    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk4nw\" (UniqueName: \"kubernetes.io/projected/70706d89-ccba-4014-b883-6fd5ff43b97a-kube-api-access-pk4nw\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:30.234341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.234278    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" event={"ID":"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d","Type":"ContainerStarted","Data":"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"}
Nov 01 00:36:30.234341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.234306    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" podUID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" containerName="authorino" containerID="cri-o://e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172" gracePeriod=30
Nov 01 00:36:30.235721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.235693    4493 generic.go:356] "Generic (PLEG): container finished" podID="70706d89-ccba-4014-b883-6fd5ff43b97a" containerID="3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0" exitCode=0
Nov 01 00:36:30.235859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.235734    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff4f9f84d-j977b"
Nov 01 00:36:30.235859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.235786    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff4f9f84d-j977b" event={"ID":"70706d89-ccba-4014-b883-6fd5ff43b97a","Type":"ContainerDied","Data":"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"}
Nov 01 00:36:30.235859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.235843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff4f9f84d-j977b" event={"ID":"70706d89-ccba-4014-b883-6fd5ff43b97a","Type":"ContainerDied","Data":"547536965a289e42f3b90da9500109b57852d19916b493778ec75f97eac6f8b9"}
Nov 01 00:36:30.236010 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.235867    4493 scope.go:122] "RemoveContainer" containerID="3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"
Nov 01 00:36:30.246257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.246155    4493 scope.go:122] "RemoveContainer" containerID="3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"
Nov 01 00:36:30.246602 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:30.246579    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0\": container with ID starting with 3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0 not found: ID does not exist" containerID="3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"
Nov 01 00:36:30.246689 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.246616    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0"} err="failed to get container status \"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0\": rpc error: code = NotFound desc = could not find container \"3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0\": container with ID starting with 3fa3f0e6968d4d555fdcf6971395c1fd7cb2798008ba9dc0282e131b092786b0 not found: ID does not exist"
Nov 01 00:36:30.349373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.348707    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" podStartSLOduration=2.654054978 podStartE2EDuration="15.348686749s" totalImagesPullingTime="12.694631771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:36:30.347828334 +0000 UTC m=+780.025415278" watchObservedRunningTime="2026-11-01 00:36:30.348686749 +0000 UTC m=+780.026273696"
Nov 01 00:36:30.458492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.458455    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:30.464648 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.464624    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ff4f9f84d-j977b"]
Nov 01 00:36:30.493555 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.493485    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:30.500845 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.500812    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70706d89-ccba-4014-b883-6fd5ff43b97a" path="/var/lib/kubelet/pods/70706d89-ccba-4014-b883-6fd5ff43b97a/volumes"
Nov 01 00:36:30.570688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.570661    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\" (UniqueName: \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\") pod \"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d\" (UID: \"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d\") "
Nov 01 00:36:30.572855 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.572833    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58" pod "b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" (UID: "b7639bfe-32aa-4cb7-b555-307a9bf0aa2d"). InnerVolumeSpecName "kube-api-access-p5l58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:36:30.671909 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:30.671878    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5l58\" (UniqueName: \"kubernetes.io/projected/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d-kube-api-access-p5l58\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:36:31.240966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.240932    4493 generic.go:356] "Generic (PLEG): container finished" podID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" containerID="e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172" exitCode=0
Nov 01 00:36:31.241184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.240981    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-slmzw"
Nov 01 00:36:31.241184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.240985    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" event={"ID":"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d","Type":"ContainerDied","Data":"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"}
Nov 01 00:36:31.241184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.241116    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-slmzw" event={"ID":"b7639bfe-32aa-4cb7-b555-307a9bf0aa2d","Type":"ContainerDied","Data":"8e357be4e6221fd7e0b9b402a17b5899d22862ce65d16ceaed161904f62a7425"}
Nov 01 00:36:31.241184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.241168    4493 scope.go:122] "RemoveContainer" containerID="e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"
Nov 01 00:36:31.249624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.249609    4493 scope.go:122] "RemoveContainer" containerID="e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"
Nov 01 00:36:31.249868 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:31.249850    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172\": container with ID starting with e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172 not found: ID does not exist" containerID="e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"
Nov 01 00:36:31.249928 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.249878    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172"} err="failed to get container status \"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172\": rpc error: code = NotFound desc = could not find container \"e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172\": container with ID starting with e314107a0d2e05d7849757935cf613e83c5e52361d2d705860a1f30f64414172 not found: ID does not exist"
Nov 01 00:36:31.409091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.409061    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:31.414823 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:31.414799    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-slmzw"]
Nov 01 00:36:32.500455 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:36:32.500424    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" path="/var/lib/kubelet/pods/b7639bfe-32aa-4cb7-b555-307a9bf0aa2d/volumes"
Nov 01 00:36:44.988341 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:36:44.988301    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:37:14.988225 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:37:14.988183    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:37:44.989050 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:37:44.988974    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:37:54.119952 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.119919    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"]
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120234    4493 cpu_manager.go:414] "containerMap: removing container" podUID="06d59e98-4a79-4acb-8743-146dd13edd4c" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120245    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d59e98-4a79-4acb-8743-146dd13edd4c" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120257    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7963a061-dbb8-4e6a-a914-83a22de62335" containerName="maas-api"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120262    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7963a061-dbb8-4e6a-a914-83a22de62335" containerName="maas-api"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120269    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120274    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120284    4493 cpu_manager.go:414] "containerMap: removing container" podUID="70706d89-ccba-4014-b883-6fd5ff43b97a" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120288    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70706d89-ccba-4014-b883-6fd5ff43b97a" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120342    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d59e98-4a79-4acb-8743-146dd13edd4c" containerName="authorino"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120349    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7963a061-dbb8-4e6a-a914-83a22de62335" containerName="maas-api"
Nov 01 00:37:54.120352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120355    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7639bfe-32aa-4cb7-b555-307a9bf0aa2d" containerName="authorino"
Nov 01 00:37:54.120681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.120362    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="70706d89-ccba-4014-b883-6fd5ff43b97a" containerName="authorino"
Nov 01 00:37:54.123361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.123346    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.225747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.225717    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 00:37:54.225903 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.225868    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 00:37:54.225945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.225908    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 00:37:54.225985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.225971    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s6ps6\""
Nov 01 00:37:54.227187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.227169    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"]
Nov 01 00:37:54.238877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.238851    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.239088 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.239060    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.239208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.239108    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.239275 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.239234    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.239405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.239349    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tj7\" (UniqueName: \"kubernetes.io/projected/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-kube-api-access-z4tj7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.239505 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.239415    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340121 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340084    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tj7\" (UniqueName: \"kubernetes.io/projected/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-kube-api-access-z4tj7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340126    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340186    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340202    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340221    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340646    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.340765 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.340679    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.342528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.342508    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.342808 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.342792    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.401433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.401370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tj7\" (UniqueName: \"kubernetes.io/projected/18a2f4eb-14ce-4b2a-9595-1df6fcf4576e-kube-api-access-z4tj7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6wztl\" (UID: \"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.435334 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.435283    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:37:54.806505 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.806473    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"]
Nov 01 00:37:54.809417 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:37:54.809388    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a2f4eb_14ce_4b2a_9595_1df6fcf4576e.slice/crio-8ef80abddb1deb9d4165f9073611e801815de85ef0ca680bb6e42e44c82c7a44 WatchSource:0}: Error finding container 8ef80abddb1deb9d4165f9073611e801815de85ef0ca680bb6e42e44c82c7a44: Status 404 returned error can't find the container with id 8ef80abddb1deb9d4165f9073611e801815de85ef0ca680bb6e42e44c82c7a44
Nov 01 00:37:54.810740 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:54.810725    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:37:55.563509 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:55.563473    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl" event={"ID":"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e","Type":"ContainerStarted","Data":"8ef80abddb1deb9d4165f9073611e801815de85ef0ca680bb6e42e44c82c7a44"}
Nov 01 00:37:57.573786 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:57.573751    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl" event={"ID":"18a2f4eb-14ce-4b2a-9595-1df6fcf4576e","Type":"ContainerStarted","Data":"8c0b87344a5cde4d1ad205b8d7823bc8e12941dc72211de058aa5cc0e58fd6c9"}
Nov 01 00:37:57.694218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:57.694156    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl" podStartSLOduration=1.356261725 podStartE2EDuration="3.694117375s" totalImagesPullingTime="2.33785565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:37:57.693799066 +0000 UTC m=+867.371386012" watchObservedRunningTime="2026-11-01 00:37:57.694117375 +0000 UTC m=+867.371704318"
Nov 01 00:37:57.710407 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:57.710383    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:37:58.435854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:37:58.435826    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:38:04.435615 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:04.435579    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:38:04.448543 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:04.448518    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:38:04.617430 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:04.617400    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6wztl"
Nov 01 00:38:04.982028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:04.982000    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"]
Nov 01 00:38:04.985359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:04.985342    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.038853 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.038745    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 00:38:05.044517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.044488    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"]
Nov 01 00:38:05.133818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.133790    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70817335-7e72-4250-b50d-acb0c8adbda5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.133983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.133834    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdg6w\" (UniqueName: \"kubernetes.io/projected/70817335-7e72-4250-b50d-acb0c8adbda5-kube-api-access-cdg6w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.133983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.133883    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.134064 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.133976    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.134064 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.134054    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.134155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.134087    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.234860 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234768    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234866    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234918    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70817335-7e72-4250-b50d-acb0c8adbda5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234943    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdg6w\" (UniqueName: \"kubernetes.io/projected/70817335-7e72-4250-b50d-acb0c8adbda5-kube-api-access-cdg6w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234963    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.234982    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.235271    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.235328    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.235489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.235434    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.237434 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.237410    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70817335-7e72-4250-b50d-acb0c8adbda5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.237619 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.237603    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70817335-7e72-4250-b50d-acb0c8adbda5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.296322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.296292    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdg6w\" (UniqueName: \"kubernetes.io/projected/70817335-7e72-4250-b50d-acb0c8adbda5-kube-api-access-cdg6w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q25ns\" (UID: \"70817335-7e72-4250-b50d-acb0c8adbda5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.298151 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.298116    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:05.669353 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:05.669274    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"]
Nov 01 00:38:05.672218 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:05.672195    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70817335_7e72_4250_b50d_acb0c8adbda5.slice/crio-114689ac515eec57b74602928c312cf35b2293069cc1438c9c86615738d2cd7e WatchSource:0}: Error finding container 114689ac515eec57b74602928c312cf35b2293069cc1438c9c86615738d2cd7e: Status 404 returned error can't find the container with id 114689ac515eec57b74602928c312cf35b2293069cc1438c9c86615738d2cd7e
Nov 01 00:38:06.615801 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:06.615765    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns" event={"ID":"70817335-7e72-4250-b50d-acb0c8adbda5","Type":"ContainerStarted","Data":"76b1021a4ca71144d8fdb0b0591163a4ea6160a002a3a2f0d517a5871e8b3c40"}
Nov 01 00:38:06.615968 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:06.615820    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns" event={"ID":"70817335-7e72-4250-b50d-acb0c8adbda5","Type":"ContainerStarted","Data":"114689ac515eec57b74602928c312cf35b2293069cc1438c9c86615738d2cd7e"}
Nov 01 00:38:06.732962 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:06.732902    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns" podStartSLOduration=1.9443157100000001 podStartE2EDuration="2.732886277s" totalImagesPullingTime="788.570567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:06.732782786 +0000 UTC m=+876.410369724" watchObservedRunningTime="2026-11-01 00:38:06.732886277 +0000 UTC m=+876.410473221"
Nov 01 00:38:07.298879 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:07.298846    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:07.702341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:07.702264    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:38:09.194644 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.194610    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"]
Nov 01 00:38:09.197936 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.197914    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.250486 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.250459    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 00:38:09.301506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.301477    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"]
Nov 01 00:38:09.369340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369308    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.369520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bljz\" (UniqueName: \"kubernetes.io/projected/7b66841e-e506-4f6f-bc1a-c79da07fbde6-kube-api-access-8bljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.369520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369431    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.369520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369469    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.369622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369525    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.369622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.369545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470154    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470203    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bljz\" (UniqueName: \"kubernetes.io/projected/7b66841e-e506-4f6f-bc1a-c79da07fbde6-kube-api-access-8bljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470226    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470243    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470265    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470282    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470656 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470599    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470656 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470631    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.470745 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.470675    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.472690 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.472673    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b66841e-e506-4f6f-bc1a-c79da07fbde6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.472957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.472936    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b66841e-e506-4f6f-bc1a-c79da07fbde6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.534023 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.533994    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bljz\" (UniqueName: \"kubernetes.io/projected/7b66841e-e506-4f6f-bc1a-c79da07fbde6-kube-api-access-8bljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r\" (UID: \"7b66841e-e506-4f6f-bc1a-c79da07fbde6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:09.809736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:09.809658    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:10.182349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:10.182323    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"]
Nov 01 00:38:10.184851 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:10.184825    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b66841e_e506_4f6f_bc1a_c79da07fbde6.slice/crio-6a0e4608a48be6658114d013859ab7e453cf6d144a351dbfd89dc9a29f04e630 WatchSource:0}: Error finding container 6a0e4608a48be6658114d013859ab7e453cf6d144a351dbfd89dc9a29f04e630: Status 404 returned error can't find the container with id 6a0e4608a48be6658114d013859ab7e453cf6d144a351dbfd89dc9a29f04e630
Nov 01 00:38:10.633391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:10.633357    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r" event={"ID":"7b66841e-e506-4f6f-bc1a-c79da07fbde6","Type":"ContainerStarted","Data":"6a0e4608a48be6658114d013859ab7e453cf6d144a351dbfd89dc9a29f04e630"}
Nov 01 00:38:11.639267 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:11.639226    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r" event={"ID":"7b66841e-e506-4f6f-bc1a-c79da07fbde6","Type":"ContainerStarted","Data":"84e0020bb246cc5fbfcbd9c89f8529515f57cddb43a2497a3de0e245152ea918"}
Nov 01 00:38:11.756874 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:11.756819    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r" podStartSLOduration=1.968347165 podStartE2EDuration="2.756800892s" totalImagesPullingTime="788.453727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:11.756482913 +0000 UTC m=+881.434069859" watchObservedRunningTime="2026-11-01 00:38:11.756800892 +0000 UTC m=+881.434387829"
Nov 01 00:38:11.809942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:11.809896    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:14.989974 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:38:14.989870    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:38:15.299186 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:15.299105    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:15.311978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:15.311951    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:15.673039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:15.672948    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q25ns"
Nov 01 00:38:16.200279 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.200248    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"]
Nov 01 00:38:16.204163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.204124    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.272945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.272912    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 00:38:16.324254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.324222    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"]
Nov 01 00:38:16.329727 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329696    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.329878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329748    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.329878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329817    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.329878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329853    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.330045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329882    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.330045 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.329990    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz26\" (UniqueName: \"kubernetes.io/projected/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-kube-api-access-nhz26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431279    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431390    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431429    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431469    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz26\" (UniqueName: \"kubernetes.io/projected/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-kube-api-access-nhz26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431501    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431694 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431548    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431870 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431843    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431904    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.431957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.431924    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.433655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.433632    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.433954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.433938    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.493290 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.493261    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz26\" (UniqueName: \"kubernetes.io/projected/775aeac2-6bb0-4e45-9019-2e09e6e8b4f5-kube-api-access-nhz26\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k\" (UID: \"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.517026 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.516997    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:16.890244 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:16.890163    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"]
Nov 01 00:38:16.892975 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:16.892950    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775aeac2_6bb0_4e45_9019_2e09e6e8b4f5.slice/crio-808cc0a69d53c4277fe2c101a95ff77d37c01918742557740afabd43d6608660 WatchSource:0}: Error finding container 808cc0a69d53c4277fe2c101a95ff77d37c01918742557740afabd43d6608660: Status 404 returned error can't find the container with id 808cc0a69d53c4277fe2c101a95ff77d37c01918742557740afabd43d6608660
Nov 01 00:38:17.671466 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:17.671436    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k" event={"ID":"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5","Type":"ContainerStarted","Data":"808cc0a69d53c4277fe2c101a95ff77d37c01918742557740afabd43d6608660"}
Nov 01 00:38:18.676884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:18.676846    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k" event={"ID":"775aeac2-6bb0-4e45-9019-2e09e6e8b4f5","Type":"ContainerStarted","Data":"d583441ed88580d6c0b49d3ba2a25ec614ef5a3e45971ec57e0fd434872913f3"}
Nov 01 00:38:18.797933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:18.797878    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k" podStartSLOduration=2.022280552 podStartE2EDuration="2.797863472s" totalImagesPullingTime="775.58292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:18.796858442 +0000 UTC m=+888.474445388" watchObservedRunningTime="2026-11-01 00:38:18.797863472 +0000 UTC m=+888.475450415"
Nov 01 00:38:19.517294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:19.517262    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:19.810077 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:19.809997    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:19.822323 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:19.822299    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:20.694779 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:20.694750    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r"
Nov 01 00:38:24.182151 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:24.182104    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:38:26.517721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:26.517683    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:26.530110 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:26.530084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:26.718237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:26.718208    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k"
Nov 01 00:38:35.613396 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.613366    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 00:38:35.617757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.617738    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.670269 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.670232    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\""
Nov 01 00:38:35.711177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711143    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711229    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711263    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711292    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711318    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711505 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711457    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711505 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711490    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711505    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs645\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.711580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.711523    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.722974 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.722944    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 00:38:35.812866 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.812834    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.812866 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.812869    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs645\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.812888    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.812916    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.812955    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813068    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813195    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813230    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813360    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813518 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813518 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813439    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813592    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.813893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.813876    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.815318 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.815299    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.816011 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.815993    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.874301 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.874222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.874457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.874328    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs645\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\") pod \"maas-default-gateway-openshift-default-67bf986f99-rswbl\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:35.929897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:35.929859    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:36.354582 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:36.354559    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 00:38:36.357018 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:36.356990    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ab1e26_cf86_4ada_9e27_848ef59d1a81.slice/crio-e30c849c148876432337e236f4fe2966d94d590ac9795a5b84295532fcbe4461 WatchSource:0}: Error finding container e30c849c148876432337e236f4fe2966d94d590ac9795a5b84295532fcbe4461: Status 404 returned error can't find the container with id e30c849c148876432337e236f4fe2966d94d590ac9795a5b84295532fcbe4461
Nov 01 00:38:36.748454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:36.748413    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" event={"ID":"c5ab1e26-cf86-4ada-9e27-848ef59d1a81","Type":"ContainerStarted","Data":"e30c849c148876432337e236f4fe2966d94d590ac9795a5b84295532fcbe4461"}
Nov 01 00:38:37.279507 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:37.279473    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:38:40.221081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:40.221054    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:38:40.221379 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:40.221120    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:38:40.221379 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:40.221177    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:38:40.768469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:40.768426    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" event={"ID":"c5ab1e26-cf86-4ada-9e27-848ef59d1a81","Type":"ContainerStarted","Data":"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"}
Nov 01 00:38:40.936506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:40.936408    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:41.002578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:41.002525    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podStartSLOduration=2.141263631 podStartE2EDuration="6.002508719s" totalImagesPullingTime="3.861245088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:40.951698171 +0000 UTC m=+910.629285116" watchObservedRunningTime="2026-11-01 00:38:41.002508719 +0000 UTC m=+910.680095720"
Nov 01 00:38:41.941824 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:41.941790    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:41.942342 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:41.941836    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:42.940272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:42.940235    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:42.940428 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:42.940292    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:43.930564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:43.930523    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:43.931182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:43.931150    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:44.940299 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:44.940263    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:44.940720 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:44.940321    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:44.988422 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:38:44.988387    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:38:45.935599 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:45.935557    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:45.935901 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:45.935869    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:46.930769 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:46.930732    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 00:38:46.931199 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:46.930797    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 00:38:48.341377 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.341330    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8587cb6b5c-nb977"]
Nov 01 00:38:48.344836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.344811    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.402568 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.402503    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8587cb6b5c-nb977"]
Nov 01 00:38:48.426937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.426905    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwhx\" (UniqueName: \"kubernetes.io/projected/808deacd-de73-48da-8649-28c36316e280-kube-api-access-8zwhx\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.426937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.426939    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808deacd-de73-48da-8649-28c36316e280-tls-cert\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.486731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.486694    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:48.487060 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.487034    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:48.487983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.487942    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 00:38:48.528588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.528550    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwhx\" (UniqueName: \"kubernetes.io/projected/808deacd-de73-48da-8649-28c36316e280-kube-api-access-8zwhx\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.528792 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.528596    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808deacd-de73-48da-8649-28c36316e280-tls-cert\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.531391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.531359    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808deacd-de73-48da-8649-28c36316e280-tls-cert\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.593769 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.593698    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwhx\" (UniqueName: \"kubernetes.io/projected/808deacd-de73-48da-8649-28c36316e280-kube-api-access-8zwhx\") pod \"authorino-8587cb6b5c-nb977\" (UID: \"808deacd-de73-48da-8649-28c36316e280\") " pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:48.656217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:48.656182    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8587cb6b5c-nb977"
Nov 01 00:38:49.026949 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:49.026925    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8587cb6b5c-nb977"]
Nov 01 00:38:49.029638 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:49.029610    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod808deacd_de73_48da_8649_28c36316e280.slice/crio-fca7a742a905b66fe550d690c4d5ac775571cf7a01b31713d11c570f84578a8f WatchSource:0}: Error finding container fca7a742a905b66fe550d690c4d5ac775571cf7a01b31713d11c570f84578a8f: Status 404 returned error can't find the container with id fca7a742a905b66fe550d690c4d5ac775571cf7a01b31713d11c570f84578a8f
Nov 01 00:38:49.805756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:49.805720    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8587cb6b5c-nb977" event={"ID":"808deacd-de73-48da-8649-28c36316e280","Type":"ContainerStarted","Data":"fca7a742a905b66fe550d690c4d5ac775571cf7a01b31713d11c570f84578a8f"}
Nov 01 00:38:52.819320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:52.819287    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8587cb6b5c-nb977" event={"ID":"808deacd-de73-48da-8649-28c36316e280","Type":"ContainerStarted","Data":"4d4707edfc7098f8d6ba584c1bcd77ce1c97f6043655b6d51875ba2ce68f1d11"}
Nov 01 00:38:52.936279 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:52.936226    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8587cb6b5c-nb977" podStartSLOduration=1.741723623 podStartE2EDuration="4.936212336s" totalImagesPullingTime="3.194488713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:52.934784562 +0000 UTC m=+922.612371507" watchObservedRunningTime="2026-11-01 00:38:52.936212336 +0000 UTC m=+922.613799281"
Nov 01 00:38:52.968574 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:52.968544    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:38:52.968770 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:52.968717    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b99d64678-t8mjf" podUID="945d3aca-dad1-4177-9190-1e07888c42c6" containerName="authorino" containerID="cri-o://06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4" gracePeriod=30
Nov 01 00:38:53.208367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.208340    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:38:53.272117 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.272086    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\" (UniqueName: \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\") pod \"945d3aca-dad1-4177-9190-1e07888c42c6\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") "
Nov 01 00:38:53.272313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.272125    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\" (UniqueName: \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\") pod \"945d3aca-dad1-4177-9190-1e07888c42c6\" (UID: \"945d3aca-dad1-4177-9190-1e07888c42c6\") "
Nov 01 00:38:53.274442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.274408    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z" pod "945d3aca-dad1-4177-9190-1e07888c42c6" (UID: "945d3aca-dad1-4177-9190-1e07888c42c6"). InnerVolumeSpecName "kube-api-access-hcf2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:38:53.283448 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.283415    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert" pod "945d3aca-dad1-4177-9190-1e07888c42c6" (UID: "945d3aca-dad1-4177-9190-1e07888c42c6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:38:53.373606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.373508    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcf2z\" (UniqueName: \"kubernetes.io/projected/945d3aca-dad1-4177-9190-1e07888c42c6-kube-api-access-hcf2z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:38:53.373606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.373549    4493 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/945d3aca-dad1-4177-9190-1e07888c42c6-tls-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:38:53.824757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.824726    4493 generic.go:356] "Generic (PLEG): container finished" podID="945d3aca-dad1-4177-9190-1e07888c42c6" containerID="06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4" exitCode=0
Nov 01 00:38:53.825204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.824775    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b99d64678-t8mjf"
Nov 01 00:38:53.825204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.824817    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b99d64678-t8mjf" event={"ID":"945d3aca-dad1-4177-9190-1e07888c42c6","Type":"ContainerDied","Data":"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"}
Nov 01 00:38:53.825204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.824863    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b99d64678-t8mjf" event={"ID":"945d3aca-dad1-4177-9190-1e07888c42c6","Type":"ContainerDied","Data":"051cf12be99dfd9ca32d1d8bdff02f8d2708d6a94db3918667799e01f9de63bb"}
Nov 01 00:38:53.825204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.824884    4493 scope.go:122] "RemoveContainer" containerID="06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"
Nov 01 00:38:53.833699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.833676    4493 scope.go:122] "RemoveContainer" containerID="06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"
Nov 01 00:38:53.834024 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:38:53.834003    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4\": container with ID starting with 06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4 not found: ID does not exist" containerID="06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"
Nov 01 00:38:53.834094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.834034    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4"} err="failed to get container status \"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4\": rpc error: code = NotFound desc = could not find container \"06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4\": container with ID starting with 06dc7d0840c6d9cc5bd8eab9d67965765d066692b7eca6c4f4742b1a4a78b6f4 not found: ID does not exist"
Nov 01 00:38:53.988270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.988238    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:38:53.992108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:53.992083    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b99d64678-t8mjf"]
Nov 01 00:38:54.501955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:54.501913    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945d3aca-dad1-4177-9190-1e07888c42c6" path="/var/lib/kubelet/pods/945d3aca-dad1-4177-9190-1e07888c42c6/volumes"
Nov 01 00:38:58.359196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.359160    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"]
Nov 01 00:38:58.359589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.359519    4493 cpu_manager.go:414] "containerMap: removing container" podUID="945d3aca-dad1-4177-9190-1e07888c42c6" containerName="authorino"
Nov 01 00:38:58.359589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.359530    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945d3aca-dad1-4177-9190-1e07888c42c6" containerName="authorino"
Nov 01 00:38:58.359589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.359581    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="945d3aca-dad1-4177-9190-1e07888c42c6" containerName="authorino"
Nov 01 00:38:58.363713 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.363695    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.424532 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.424492    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2699z\" (UniqueName: \"kubernetes.io/projected/2795bb4c-0811-45a4-b027-d436e65d481b-kube-api-access-2699z\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.424715 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.424567    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795bb4c-0811-45a4-b027-d436e65d481b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.466143 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.466093    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rg5z4\""
Nov 01 00:38:58.466143 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.466101    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 00:38:58.467516 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.467493    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"]
Nov 01 00:38:58.518204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.518167    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-jdwxc"]
Nov 01 00:38:58.521733 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.521713    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.525016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.524986    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795bb4c-0811-45a4-b027-d436e65d481b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.525167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.525089    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2699z\" (UniqueName: \"kubernetes.io/projected/2795bb4c-0811-45a4-b027-d436e65d481b-kube-api-access-2699z\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.525748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.525725    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795bb4c-0811-45a4-b027-d436e65d481b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.566114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.566077    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-jdwxc"]
Nov 01 00:38:58.589293 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.589261    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2699z\" (UniqueName: \"kubernetes.io/projected/2795bb4c-0811-45a4-b027-d436e65d481b-kube-api-access-2699z\") pod \"payload-pre-processing-7b98bb678c-cr9md\" (UID: \"2795bb4c-0811-45a4-b027-d436e65d481b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.626314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.626230    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2595c2-9fd3-4764-a501-e137bbabf621-plugins-config-volume\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.626314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.626280    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn9n\" (UniqueName: \"kubernetes.io/projected/2f2595c2-9fd3-4764-a501-e137bbabf621-kube-api-access-lpn9n\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.676764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.676732    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:38:58.728046 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.727999    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2595c2-9fd3-4764-a501-e137bbabf621-plugins-config-volume\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.728264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.728057    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn9n\" (UniqueName: \"kubernetes.io/projected/2f2595c2-9fd3-4764-a501-e137bbabf621-kube-api-access-lpn9n\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.728807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.728780    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2595c2-9fd3-4764-a501-e137bbabf621-plugins-config-volume\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.790879 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.790841    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn9n\" (UniqueName: \"kubernetes.io/projected/2f2595c2-9fd3-4764-a501-e137bbabf621-kube-api-access-lpn9n\") pod \"payload-processing-7bb9489676-jdwxc\" (UID: \"2f2595c2-9fd3-4764-a501-e137bbabf621\") " pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:58.835005 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:58.834970    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:38:59.054889 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:59.054869    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"]
Nov 01 00:38:59.057339 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:59.057308    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2795bb4c_0811_45a4_b027_d436e65d481b.slice/crio-ccc896f552de5ad2af899834419a3e69d8078604a939fd699faf5e291dc6a683 WatchSource:0}: Error finding container ccc896f552de5ad2af899834419a3e69d8078604a939fd699faf5e291dc6a683: Status 404 returned error can't find the container with id ccc896f552de5ad2af899834419a3e69d8078604a939fd699faf5e291dc6a683
Nov 01 00:38:59.212525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:59.212450    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-jdwxc"]
Nov 01 00:38:59.216051 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:38:59.216021    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2595c2_9fd3_4764_a501_e137bbabf621.slice/crio-54bc85f01f435e6da296fe0d8deae2b3088385efe85eab069f7ef453ed291e5e WatchSource:0}: Error finding container 54bc85f01f435e6da296fe0d8deae2b3088385efe85eab069f7ef453ed291e5e: Status 404 returned error can't find the container with id 54bc85f01f435e6da296fe0d8deae2b3088385efe85eab069f7ef453ed291e5e
Nov 01 00:38:59.868467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:59.868422    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md" event={"ID":"2795bb4c-0811-45a4-b027-d436e65d481b","Type":"ContainerStarted","Data":"ccc896f552de5ad2af899834419a3e69d8078604a939fd699faf5e291dc6a683"}
Nov 01 00:38:59.869830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:38:59.869801    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc" event={"ID":"2f2595c2-9fd3-4764-a501-e137bbabf621","Type":"ContainerStarted","Data":"54bc85f01f435e6da296fe0d8deae2b3088385efe85eab069f7ef453ed291e5e"}
Nov 01 00:39:02.883984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:02.883952    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md" event={"ID":"2795bb4c-0811-45a4-b027-d436e65d481b","Type":"ContainerStarted","Data":"4090eeb5c646f6d2e5e0fdb3812ecb08027d4d1741fa0de8f6b04f49d9a0508c"}
Nov 01 00:39:02.884484 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:02.884021    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:39:02.885550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:02.885526    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc" event={"ID":"2f2595c2-9fd3-4764-a501-e137bbabf621","Type":"ContainerStarted","Data":"c6453ba826914b24e3b3d1fa5cb7802142f5d83b314fd610c6d4d2f47097b597"}
Nov 01 00:39:02.885645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:02.885568    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:39:03.000174 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:03.000095    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md" podStartSLOduration=1.9607681989999999 podStartE2EDuration="5.000077647s" totalImagesPullingTime="3.039309448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:39:02.999580673 +0000 UTC m=+932.677167620" watchObservedRunningTime="2026-11-01 00:39:03.000077647 +0000 UTC m=+932.677664590"
Nov 01 00:39:03.116284 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:03.116216    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc" podStartSLOduration=2.273795775 podStartE2EDuration="5.116197011s" totalImagesPullingTime="2.842401236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:39:03.11536298 +0000 UTC m=+932.792949926" watchObservedRunningTime="2026-11-01 00:39:03.116197011 +0000 UTC m=+932.793783958"
Nov 01 00:39:05.852076 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.852041    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:39:05.855548 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.855531    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.959971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.959935    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:39:05.995996 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.995961    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.995996 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.995994    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996026    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996185    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996223    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996250    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996370    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:05.996440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:05.996432    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2bw\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097229 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097195    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097229 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097229    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097251    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097290    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097309    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097386    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2bw\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097823 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097652    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097823 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097699    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097823 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.097966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.097860    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.098220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.098118    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.099783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.099761    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.099867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.099812    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.158525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.158457    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.158631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.158568    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2bw\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-86c2s\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.168454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.168428    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:06.596377 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.596352    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:39:06.598585 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:39:06.598557    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87ed863_cd59_4d89_8d23_a6be691bff76.slice/crio-77681753903a851fa5abb239e58836c7997278febca2b3cc8fc79a39f83357f7 WatchSource:0}: Error finding container 77681753903a851fa5abb239e58836c7997278febca2b3cc8fc79a39f83357f7: Status 404 returned error can't find the container with id 77681753903a851fa5abb239e58836c7997278febca2b3cc8fc79a39f83357f7
Nov 01 00:39:06.600726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.600700    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:39:06.600810 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.600763    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:39:06.600810 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.600798    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:39:06.904550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.904455    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" event={"ID":"a87ed863-cd59-4d89-8d23-a6be691bff76","Type":"ContainerStarted","Data":"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"}
Nov 01 00:39:06.904550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:06.904525    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" event={"ID":"a87ed863-cd59-4d89-8d23-a6be691bff76","Type":"ContainerStarted","Data":"77681753903a851fa5abb239e58836c7997278febca2b3cc8fc79a39f83357f7"}
Nov 01 00:39:07.032675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:07.030668    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podStartSLOduration=2.030650358 podStartE2EDuration="2.030650358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:39:07.025927297 +0000 UTC m=+936.703514274" watchObservedRunningTime="2026-11-01 00:39:07.030650358 +0000 UTC m=+936.708237320"
Nov 01 00:39:07.169683 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:07.169587    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:07.170784 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:07.170759    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:07.170901 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:07.170817    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:08.169884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:08.169846    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:08.170327 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:08.169900    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:09.168880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:09.168840    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:09.169061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:09.168898    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:10.169560 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:10.169520    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:10.169955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:10.169580    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:11.169294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:11.169260    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:11.169533 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:11.169312    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:12.169576 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:12.169494    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:12.169576 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:12.169558    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:13.169910 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:13.169869    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-86c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 00:39:13.170351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:13.169944    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 00:39:13.892276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:13.892244    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-jdwxc"
Nov 01 00:39:13.892443 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:13.892308    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cr9md"
Nov 01 00:39:14.904464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:14.904429    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:14.904976 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:14.904951    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:14.905775 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:39:14.905754    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:39:14.989011 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:39:14.988979    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:39:44.988701 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:39:44.988664    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:40:00.662563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.662528    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 00:40:00.666462 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.666443    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.722022 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.721994    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-30fb19-openshift-default-dockercfg-tm5sb\""
Nov 01 00:40:00.770243 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.770215    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 00:40:00.797964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.797863    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.797964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.797924    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.797991    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798019    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmb5\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798045    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798179    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798217    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798254    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.798477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.798279    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899272    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899310    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899332    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899352    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899399    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899421    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899447    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899472    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmb5\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.899863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.899734    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.900048 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.900022    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.900166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.900079    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.900262 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.900244    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.900377 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.900356    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.901937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.901916    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.902058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.902012    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.963512 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.963451    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.964362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.964339    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmb5\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:00.986796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:00.986768    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:01.442931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.442768    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 00:40:01.446012 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:01.445982    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3d0a61_b39e_4f6d_8d12_db11cfdfdc8c.slice/crio-e46bd1d24684fd3d79c0e7d2d4b462117e83c2854282308077e9d0b591171e32 WatchSource:0}: Error finding container e46bd1d24684fd3d79c0e7d2d4b462117e83c2854282308077e9d0b591171e32: Status 404 returned error can't find the container with id e46bd1d24684fd3d79c0e7d2d4b462117e83c2854282308077e9d0b591171e32
Nov 01 00:40:01.448623 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.448593    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:01.448682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.448657    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:01.448722 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.448683    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:01.610201 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.610173    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:40:01.613877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.613855    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.670174 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.666962    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-32439f-openshift-default-dockercfg-n9rr6\""
Nov 01 00:40:01.677072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.676983    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:40:01.708017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.707929    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.707983    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708013    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708105    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708192    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708224    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nmc\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708268    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708293    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.708563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.708354    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809739    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809765    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809809    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809840    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nmc\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809904    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809926    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.809991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.809982    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.810372 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.810181    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.810372 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.810241    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.810479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.810431    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.810805 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.810780    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.811008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.810989    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.812375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.812345    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.812489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.812474    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.871208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.871123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.871395 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.871286    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nmc\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:01.927005 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:01.926969    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:02.160556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.160520    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerStarted","Data":"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"}
Nov 01 00:40:02.160705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.160578    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerStarted","Data":"e46bd1d24684fd3d79c0e7d2d4b462117e83c2854282308077e9d0b591171e32"}
Nov 01 00:40:02.272979 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.272885    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 00:40:02.277624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.277600    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.281006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.280963    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podStartSLOduration=2.280950972 podStartE2EDuration="2.280950972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:02.280561828 +0000 UTC m=+991.958148773" watchObservedRunningTime="2026-11-01 00:40:02.280950972 +0000 UTC m=+991.958537917"
Nov 01 00:40:02.331748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.331717    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4143b7-openshift-default-dockercfg-fhj7l\""
Nov 01 00:40:02.337916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.337889    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 00:40:02.365014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.364829    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:40:02.368155 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:02.368091    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55739022_d29a_4b36_ad08_001f97401bd9.slice/crio-cf2174d4a08f33e882272680610d1b47fee030378a1d26a4a9bb8f48df3f4731 WatchSource:0}: Error finding container cf2174d4a08f33e882272680610d1b47fee030378a1d26a4a9bb8f48df3f4731: Status 404 returned error can't find the container with id cf2174d4a08f33e882272680610d1b47fee030378a1d26a4a9bb8f48df3f4731
Nov 01 00:40:02.370681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.370647    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:02.370768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.370716    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:02.370768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.370744    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:02.414609 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414578    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.414745 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414653    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.414745 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414719    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.414863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414745    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.414863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414787    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.414863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414812    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.415014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414888    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.415014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414919    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.415014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.414949    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbkt\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516478 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516439    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516498    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516569    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516604    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516630    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516666    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516708    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbkt\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516763    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.516925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516820    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.517079 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.516980    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.517079 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.517036    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.517223 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.517172    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.517278 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.517255    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.517356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.517321    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.519396 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.519374    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.519634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.519614    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.579667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.579589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.579667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.579653    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbkt\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.591565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.591536    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:02.987207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.987169    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:02.992154 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:02.992116    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:03.027066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.026918    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 00:40:03.030287 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:03.030261    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b84f89_813b_451d_9255_e0415d0dc81e.slice/crio-2f7c762a798a2a5509269ac3edf1008b9eefcac7981ca39502d358c714496181 WatchSource:0}: Error finding container 2f7c762a798a2a5509269ac3edf1008b9eefcac7981ca39502d358c714496181: Status 404 returned error can't find the container with id 2f7c762a798a2a5509269ac3edf1008b9eefcac7981ca39502d358c714496181
Nov 01 00:40:03.032433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.032402    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:03.032569 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.032480    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:03.032569 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.032523    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:03.167957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.167921    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerStarted","Data":"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"}
Nov 01 00:40:03.168160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.167977    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerStarted","Data":"2f7c762a798a2a5509269ac3edf1008b9eefcac7981ca39502d358c714496181"}
Nov 01 00:40:03.169551 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.169526    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerStarted","Data":"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"}
Nov 01 00:40:03.169655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.169570    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerStarted","Data":"cf2174d4a08f33e882272680610d1b47fee030378a1d26a4a9bb8f48df3f4731"}
Nov 01 00:40:03.169744 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.169727    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:03.170516 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.170499    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:03.290433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.290326    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podStartSLOduration=1.29030621 podStartE2EDuration="1.29030621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:03.289016756 +0000 UTC m=+992.966603716" watchObservedRunningTime="2026-11-01 00:40:03.29030621 +0000 UTC m=+992.967893157"
Nov 01 00:40:03.542775 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.542662    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podStartSLOduration=2.542641679 podStartE2EDuration="2.542641679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:03.539952276 +0000 UTC m=+993.217539222" watchObservedRunningTime="2026-11-01 00:40:03.542641679 +0000 UTC m=+993.220228627"
Nov 01 00:40:03.591848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.591812    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:03.927551 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.927471    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:03.932485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:03.932463    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:04.174016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:04.173980    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:04.175162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:04.175113    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:40:04.598459 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:04.598396    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:05.177739 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:05.177711    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:05.178711 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:05.178692    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:11.626633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.626596    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 00:40:11.630343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.630324    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.682848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.682817    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-948e22\""
Nov 01 00:40:11.751184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.738504    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-948e22-dockercfg-qhblk\""
Nov 01 00:40:11.751184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.741396    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 00:40:11.752320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.750231    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.752464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.752381    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848s5\" (UniqueName: \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.784669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.784638    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 00:40:11.788114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.788092    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.831333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.831299    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 00:40:11.853076 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.853037    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkzl\" (UniqueName: \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.853298 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.853083    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.853298 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.853252    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848s5\" (UniqueName: \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.853412 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.853322    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.854264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.854241    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.914754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.914688    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848s5\" (UniqueName: \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\") pod \"payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.943097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.943066    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:11.954498 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.954472    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkzl\" (UniqueName: \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.954565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.954520    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.955169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.955151    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:11.967080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.967052    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 00:40:11.971529 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:11.971511    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.015800 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.015723    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkzl\" (UniqueName: \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\") pod \"payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:12.025177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.024413    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 00:40:12.025177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.024413    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-22d7b2-serving-cert\""
Nov 01 00:40:12.025177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.024523    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 00:40:12.025177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.024536    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 00:40:12.025177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.024533    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jq2v\""
Nov 01 00:40:12.031843 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.031814    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 00:40:12.055681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.055647    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.055825 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.055689    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.055825 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.055799    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.055893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.055838    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s87b7\" (UniqueName: \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.101639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.101611    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:12.156989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.156681    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.156989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.156755    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.156989 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:12.156967    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-serving-cert: secret "maas-api-e2e-worker-w6-22d7b2-serving-cert" not found
Nov 01 00:40:12.157445 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:12.157050    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls podName:cb271338-c36e-44a4-93ed-08a1a9195a74 nodeName:}" failed. No retries permitted until 2026-11-01 00:40:12.657028167 +0000 UTC m=+1002.334615110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls") pod "maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" (UID: "cb271338-c36e-44a4-93ed-08a1a9195a74") : secret "maas-api-e2e-worker-w6-22d7b2-serving-cert" not found
Nov 01 00:40:12.157445 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.157396    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.157645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.157454    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s87b7\" (UniqueName: \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.158210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.158194    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.161168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.160694    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.229911 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.229882    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s87b7\" (UniqueName: \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.317633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.317609    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 00:40:12.320112 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:12.320087    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4261e9a3_3a92_4d5b_bd4b_3723857d3302.slice/crio-4d076aa067089b649e8b84682252972bacf280bc91fa324b93f2c97a2f9d0556 WatchSource:0}: Error finding container 4d076aa067089b649e8b84682252972bacf280bc91fa324b93f2c97a2f9d0556: Status 404 returned error can't find the container with id 4d076aa067089b649e8b84682252972bacf280bc91fa324b93f2c97a2f9d0556
Nov 01 00:40:12.477178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.477153    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 00:40:12.479645 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:12.479614    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf86aff_5f16_4e51_bf5c_df0e5f75b20b.slice/crio-9f4998f7eae8abca804651b314ce4b12958eba422bd02ae1aa828bf38c62b69b WatchSource:0}: Error finding container 9f4998f7eae8abca804651b314ce4b12958eba422bd02ae1aa828bf38c62b69b: Status 404 returned error can't find the container with id 9f4998f7eae8abca804651b314ce4b12958eba422bd02ae1aa828bf38c62b69b
Nov 01 00:40:12.663761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.663727    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.666380 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.666355    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") pod \"maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:12.886814 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:12.886739    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:13.216601 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.216514    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" event={"ID":"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b","Type":"ContainerStarted","Data":"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"}
Nov 01 00:40:13.216601 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.216579    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" event={"ID":"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b","Type":"ContainerStarted","Data":"9f4998f7eae8abca804651b314ce4b12958eba422bd02ae1aa828bf38c62b69b"}
Nov 01 00:40:13.216815 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.216634    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:13.218184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.218157    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" event={"ID":"4261e9a3-3a92-4d5b-bd4b-3723857d3302","Type":"ContainerStarted","Data":"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"}
Nov 01 00:40:13.218303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.218196    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" event={"ID":"4261e9a3-3a92-4d5b-bd4b-3723857d3302","Type":"ContainerStarted","Data":"4d076aa067089b649e8b84682252972bacf280bc91fa324b93f2c97a2f9d0556"}
Nov 01 00:40:13.218303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.218241    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:13.262693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.262586    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 00:40:13.265217 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:13.265192    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb271338_c36e_44a4_93ed_08a1a9195a74.slice/crio-8742f32ec4833c3c0ba64604a004e4e09bf86bde594bbefda620ee46637a8c3f WatchSource:0}: Error finding container 8742f32ec4833c3c0ba64604a004e4e09bf86bde594bbefda620ee46637a8c3f: Status 404 returned error can't find the container with id 8742f32ec4833c3c0ba64604a004e4e09bf86bde594bbefda620ee46637a8c3f
Nov 01 00:40:13.334281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:13.334179    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" podStartSLOduration=2.334164976 podStartE2EDuration="2.334164976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:13.333519064 +0000 UTC m=+1003.011106001" watchObservedRunningTime="2026-11-01 00:40:13.334164976 +0000 UTC m=+1003.011751921"
Nov 01 00:40:14.224674 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:14.224646    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" event={"ID":"cb271338-c36e-44a4-93ed-08a1a9195a74","Type":"ContainerStarted","Data":"8742f32ec4833c3c0ba64604a004e4e09bf86bde594bbefda620ee46637a8c3f"}
Nov 01 00:40:14.989668 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:14.989632    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:40:15.230859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:15.230829    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" event={"ID":"cb271338-c36e-44a4-93ed-08a1a9195a74","Type":"ContainerStarted","Data":"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"}
Nov 01 00:40:15.231285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:15.230943    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:15.351874 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:15.351766    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" podStartSLOduration=4.351753582 podStartE2EDuration="4.351753582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:13.451986797 +0000 UTC m=+1003.129573746" watchObservedRunningTime="2026-11-01 00:40:15.351753582 +0000 UTC m=+1005.029340530"
Nov 01 00:40:15.352145 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:15.352111    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" podStartSLOduration=3.445227288 podStartE2EDuration="4.352105023s" totalImagesPullingTime="906.877735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:15.351448976 +0000 UTC m=+1005.029035920" watchObservedRunningTime="2026-11-01 00:40:15.352105023 +0000 UTC m=+1005.029691968"
Nov 01 00:40:19.138955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.138924    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 00:40:19.142688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.142672    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.195071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.195043    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-30fb19\""
Nov 01 00:40:19.231863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.231828    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.231863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.231862    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjbz\" (UniqueName: \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.244932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.244904    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-30fb19-dockercfg-2xv2v\""
Nov 01 00:40:19.246937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.246915    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 00:40:19.333454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.333422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.333454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.333455    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjbz\" (UniqueName: \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.334194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.334177    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.394650 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.394583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjbz\" (UniqueName: \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\") pod \"payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.456221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.456198    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:19.827423 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:19.827398    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 00:40:19.829616 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:19.829593    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f67a0cc_b273_4635_81e9_98fcb4da881c.slice/crio-d2a12f73a82d434bf4c25b655861d5a89cd94cb6c2f0108c9bf42e36976753e9 WatchSource:0}: Error finding container d2a12f73a82d434bf4c25b655861d5a89cd94cb6c2f0108c9bf42e36976753e9: Status 404 returned error can't find the container with id d2a12f73a82d434bf4c25b655861d5a89cd94cb6c2f0108c9bf42e36976753e9
Nov 01 00:40:20.254954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:20.254916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" event={"ID":"0f67a0cc-b273-4635-81e9-98fcb4da881c","Type":"ContainerStarted","Data":"3f5e55663f567735818d7b99478a7e9eefe8bbe0e9e7d97c135441e3aadd6167"}
Nov 01 00:40:20.255400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:20.255014    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" event={"ID":"0f67a0cc-b273-4635-81e9-98fcb4da881c","Type":"ContainerStarted","Data":"d2a12f73a82d434bf4c25b655861d5a89cd94cb6c2f0108c9bf42e36976753e9"}
Nov 01 00:40:20.255400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:20.255043    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:20.373622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:20.373556    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" podStartSLOduration=1.373537894 podStartE2EDuration="1.373537894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:20.37158325 +0000 UTC m=+1010.049170202" watchObservedRunningTime="2026-11-01 00:40:20.373537894 +0000 UTC m=+1010.051124840"
Nov 01 00:40:20.390543 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:20.390509    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:40:21.240785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.240759    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 00:40:21.482347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.482313    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 00:40:21.485752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.485736    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.543660 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.543611    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4143b7\""
Nov 01 00:40:21.543962 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.543779    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4143b7-dockercfg-dpvjd\""
Nov 01 00:40:21.546111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.546081    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 00:40:21.555101 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.555042    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.555938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.555808    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llq8c\" (UniqueName: \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.657374 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.657339    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llq8c\" (UniqueName: \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.657551 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.657422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.658062 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.658046    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.718699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.718667    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llq8c\" (UniqueName: \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\") pod \"payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:21.799039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:21.798961    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:22.171917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:22.171848    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 00:40:22.174705 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:22.174677    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbc4516_586a_4530_811e_689ce371b521.slice/crio-1606eecee975f5d4ac9c9b418e7c33e13ec68fe77b6af942326d5e5d370db57c WatchSource:0}: Error finding container 1606eecee975f5d4ac9c9b418e7c33e13ec68fe77b6af942326d5e5d370db57c: Status 404 returned error can't find the container with id 1606eecee975f5d4ac9c9b418e7c33e13ec68fe77b6af942326d5e5d370db57c
Nov 01 00:40:22.272290 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:22.272253    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" event={"ID":"7dbc4516-586a-4530-811e-689ce371b521","Type":"ContainerStarted","Data":"86462206f67bc685056c3476fcf92c37c28f45b0b71620ebbd1a6f8b15061c2f"}
Nov 01 00:40:22.272412 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:22.272345    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" event={"ID":"7dbc4516-586a-4530-811e-689ce371b521","Type":"ContainerStarted","Data":"1606eecee975f5d4ac9c9b418e7c33e13ec68fe77b6af942326d5e5d370db57c"}
Nov 01 00:40:22.272475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:22.272435    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:22.389433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:22.389379    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" podStartSLOduration=1.389365361 podStartE2EDuration="1.389365361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:22.387371621 +0000 UTC m=+1012.064958578" watchObservedRunningTime="2026-11-01 00:40:22.389365361 +0000 UTC m=+1012.066952307"
Nov 01 00:40:24.227856 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:24.227827    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 00:40:24.228354 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:24.227888    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 00:40:26.624091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.624055    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 00:40:26.627898 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.627878    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.713788 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.713744    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.714127 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.714103    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpzd\" (UniqueName: \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.730347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.730310    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5a9c19-dockercfg-mjgf6\""
Nov 01 00:40:26.730347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.730327    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5a9c19\""
Nov 01 00:40:26.732457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.732432    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 00:40:26.815378 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.815343    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.815540 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.815386    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpzd\" (UniqueName: \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.816044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.816028    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.876933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.876871    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpzd\" (UniqueName: \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\") pod \"payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:26.940543 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:26.940513    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:27.316636 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:27.316611    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 00:40:27.318352 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:27.318327    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd366c6_e3f4_44c7_b6dc_9894ba1d705e.slice/crio-a60eb5cf36eb3a26602ef950c6d14468bc7132616c7a06a594940e3279499e3d WatchSource:0}: Error finding container a60eb5cf36eb3a26602ef950c6d14468bc7132616c7a06a594940e3279499e3d: Status 404 returned error can't find the container with id a60eb5cf36eb3a26602ef950c6d14468bc7132616c7a06a594940e3279499e3d
Nov 01 00:40:28.026747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.026717    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 00:40:28.030338 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.030322    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.082693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.082662    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-4143b7-serving-cert\""
Nov 01 00:40:28.129953 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.129916    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.130097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.129975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.130097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.130050    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f82z\" (UniqueName: \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.130097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.130089    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.134586 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.134567    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 00:40:28.231286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.231252    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.231481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.231338    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.231481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.231368    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f82z\" (UniqueName: \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.231481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.231411    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.231716 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:28.231507    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-serving-cert: secret "maas-api-e2e-worker-w2-4143b7-serving-cert" not found
Nov 01 00:40:28.231716 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:28.231607    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls podName:c02cb64a-b4ab-4b3b-ab40-9f7ac7413992 nodeName:}" failed. No retries permitted until 2026-11-01 00:40:28.731586461 +0000 UTC m=+1018.409173412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls") pod "maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" (UID: "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992") : secret "maas-api-e2e-worker-w2-4143b7-serving-cert" not found
Nov 01 00:40:28.232096 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.232076    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.233777 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.233760    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.296971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.293966    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f82z\" (UniqueName: \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.304329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.304286    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" event={"ID":"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e","Type":"ContainerStarted","Data":"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"}
Nov 01 00:40:28.304477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.304354    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" event={"ID":"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e","Type":"ContainerStarted","Data":"a60eb5cf36eb3a26602ef950c6d14468bc7132616c7a06a594940e3279499e3d"}
Nov 01 00:40:28.305165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.305123    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:28.422921 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.422861    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" podStartSLOduration=2.422843323 podStartE2EDuration="2.422843323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:28.422790894 +0000 UTC m=+1018.100377838" watchObservedRunningTime="2026-11-01 00:40:28.422843323 +0000 UTC m=+1018.100430268"
Nov 01 00:40:28.740610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.740571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.743929 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.743905    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:28.942520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:28.942487    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:29.370437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:29.370412    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 00:40:29.373095 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:29.373061    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02cb64a_b4ab_4b3b_ab40_9f7ac7413992.slice/crio-514634266bdd10d4754929c9a3d6dd24d10864fdac31ddd86eed18c89dd64093 WatchSource:0}: Error finding container 514634266bdd10d4754929c9a3d6dd24d10864fdac31ddd86eed18c89dd64093: Status 404 returned error can't find the container with id 514634266bdd10d4754929c9a3d6dd24d10864fdac31ddd86eed18c89dd64093
Nov 01 00:40:29.597464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:29.597428    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:40:30.322360 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:30.322313    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" event={"ID":"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992","Type":"ContainerStarted","Data":"514634266bdd10d4754929c9a3d6dd24d10864fdac31ddd86eed18c89dd64093"}
Nov 01 00:40:31.263799 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:31.263773    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 00:40:31.344177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:31.344126    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" event={"ID":"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992","Type":"ContainerStarted","Data":"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"}
Nov 01 00:40:31.344315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:31.344209    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:31.542517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:31.542396    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" podStartSLOduration=3.674547895 podStartE2EDuration="4.542375429s" totalImagesPullingTime="867.827534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:31.539596778 +0000 UTC m=+1021.217183716" watchObservedRunningTime="2026-11-01 00:40:31.542375429 +0000 UTC m=+1021.219962375"
Nov 01 00:40:33.279017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:33.278987    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 00:40:37.354468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:37.354437    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 00:40:39.383503 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.383422    4493 generic.go:356] "Generic (PLEG): container finished" podID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerID="055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62" exitCode=0
Nov 01 00:40:39.383902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.383500    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerDied","Data":"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"}
Nov 01 00:40:39.383902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.383866    4493 scope.go:122] "RemoveContainer" containerID="055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"
Nov 01 00:40:39.385063 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.385035    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:39.385180 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.385101    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:39.385180 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:39.385155    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:40.339373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:40.339342    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 00:40:40.391639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:40.391601    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerStarted","Data":"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"}
Nov 01 00:40:40.987886 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:40.987854    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:40.988060 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:40.987902    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 00:40:40.988060 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:40.987952    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 00:40:41.988112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:41.988021    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 00:40:41.988112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:41.988084    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 00:40:42.987397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:42.987364    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 00:40:42.987576 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:42.987418    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 00:40:43.987664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:43.987625    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 00:40:43.988179 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:43.987676    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 00:40:44.987885 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:44.987855    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 00:40:44.988284 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:44.987905    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 00:40:44.991877 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:44.991853    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:40:45.991634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:45.991605    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:45.992005 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:45.991867    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:45.992533 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:45.992519    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 00:40:48.829087 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.829057    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 00:40:48.832491 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.832475    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:48.890652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.890623    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-948e22-serving-cert\""
Nov 01 00:40:48.940054 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.940025    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 00:40:48.972513 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.972452    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4666\" (UniqueName: \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:48.972865 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.972839    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:48.972973 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.972882    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:48.972973 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:48.972923    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.073752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.073711    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4666\" (UniqueName: \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.073752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.073758    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.074041 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.073780    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.074041 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.073816    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.074041 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:49.073918    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-serving-cert: secret "maas-api-e2e-worker-w5-948e22-serving-cert" not found
Nov 01 00:40:49.074041 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:40:49.074003    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls podName:6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1 nodeName:}" failed. No retries permitted until 2026-11-01 00:40:49.573980438 +0000 UTC m=+1039.251567374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls") pod "maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" (UID: "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1") : secret "maas-api-e2e-worker-w5-948e22-serving-cert" not found
Nov 01 00:40:49.074598 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.074575    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.094845 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.091588    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.139074 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.139043    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4666\" (UniqueName: \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.579669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.579642    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.581997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.581977    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:49.745262 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:49.745234    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:50.118682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.118654    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 00:40:50.120690 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:40:50.120665    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6e4db4_7e1b_4837_8fac_6c27f7e99fe1.slice/crio-2ba30c4d233767476e07666d83589eb875d3984032cbd26b85e8e5ea9001c137 WatchSource:0}: Error finding container 2ba30c4d233767476e07666d83589eb875d3984032cbd26b85e8e5ea9001c137: Status 404 returned error can't find the container with id 2ba30c4d233767476e07666d83589eb875d3984032cbd26b85e8e5ea9001c137
Nov 01 00:40:50.179464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.179429    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:50.179607 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.179480    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:50.442345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.442256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" event={"ID":"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1","Type":"ContainerStarted","Data":"2ba30c4d233767476e07666d83589eb875d3984032cbd26b85e8e5ea9001c137"}
Nov 01 00:40:50.443812 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.443787    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerID="dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887" exitCode=0
Nov 01 00:40:50.443969 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.443845    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerDied","Data":"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"}
Nov 01 00:40:50.444283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.444268    4493 scope.go:122] "RemoveContainer" containerID="dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"
Nov 01 00:40:50.445588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.445561    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:50.445659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.445648    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:50.445696 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:50.445682    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:40:51.182864 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.182828    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:40:51.450269 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.450163    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" event={"ID":"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1","Type":"ContainerStarted","Data":"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"}
Nov 01 00:40:51.450269 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.450242    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:40:51.452163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.452120    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerStarted","Data":"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"}
Nov 01 00:40:51.569527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.569459    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" podStartSLOduration=2.684886143 podStartE2EDuration="3.569440047s" totalImagesPullingTime="884.553904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:51.568103933 +0000 UTC m=+1041.245690893" watchObservedRunningTime="2026-11-01 00:40:51.569440047 +0000 UTC m=+1041.247026993"
Nov 01 00:40:51.592027 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.591994    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:51.592345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.592325    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:51.592400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:51.592371    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:52.592993 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:52.592952    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:52.593405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:52.593033    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:53.592214 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:53.592177    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:53.592393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:53.592262    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:54.592933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:54.592899    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:54.593353 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:54.592954    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:55.592563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:55.592528    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 00:40:55.592741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:55.592579    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 00:40:56.645059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:56.645033    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:56.646586 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:56.646552    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:56.652269 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:56.652248    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 00:40:57.472647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:40:57.472612    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 00:41:00.274925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.274884    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:41:00.515382 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.515350    4493 generic.go:356] "Generic (PLEG): container finished" podID="55739022-d29a-4b36-ad08-001f97401bd9" containerID="0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c" exitCode=0
Nov 01 00:41:00.515612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.515424    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerDied","Data":"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"}
Nov 01 00:41:00.515824 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.515809    4493 scope.go:122] "RemoveContainer" containerID="0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"
Nov 01 00:41:00.517295 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.517267    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:41:00.517368 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.517351    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:41:00.517415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:00.517400    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:41:01.540787 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:01.540745    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerStarted","Data":"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"}
Nov 01 00:41:01.934587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:01.934496    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:41:01.939291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:01.939258    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 00:41:01.939535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:01.939507    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 00:41:02.935159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:02.927603    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 00:41:02.935159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:02.927666    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 00:41:03.935159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:03.931607    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 00:41:03.935159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:03.931668    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 00:41:04.174086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:04.174049    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:41:04.939157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:04.938583    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 00:41:04.939157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:04.938648    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 00:41:05.935156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:05.934560    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 00:41:05.935156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:05.934621    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 00:41:06.932250 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:06.932220    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:41:06.933246 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:06.933228    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:41:14.318642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:14.317480    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:41:14.989742 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:41:14.989711    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:41:34.905361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:34.905313    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:41:37.798529 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.798491    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 00:41:37.801074 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.801056    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.856583 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.856556    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"openshift-service-ca.crt\""
Nov 01 00:41:37.866966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.866941    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"kube-root-ca.crt\""
Nov 01 00:41:37.904198 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.904160    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"simulator-w4-43a0cf-kserve-self-signed-certs\""
Nov 01 00:41:37.904385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.904335    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"default-dockercfg-l7qsj\""
Nov 01 00:41:37.906288 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.906262    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 00:41:37.931976 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.931941    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmm2t\" (UniqueName: \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.932181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.931989    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.932181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.932040    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.932181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.932065    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.932181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.932110    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:37.932181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:37.932168    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033020    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033056    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033191    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033360 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033337    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmm2t\" (UniqueName: \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033399 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033369    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033577    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.033675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.033605    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.035506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.035484    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.035979 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.035959    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.095070 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.094994    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmm2t\" (UniqueName: \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\") pod \"simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") " pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.111865 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.111832    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:38.505971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.505947    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 00:41:38.731667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.731627    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" event={"ID":"2e0bc473-76d4-4b7e-bec4-32930c9f8656","Type":"ContainerStarted","Data":"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"}
Nov 01 00:41:38.731667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.731674    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" event={"ID":"2e0bc473-76d4-4b7e-bec4-32930c9f8656","Type":"ContainerStarted","Data":"0ae73dc612351b19f9a3b1d00a932fcd90844df177952d09be7761b86803bbf0"}
Nov 01 00:41:38.862628 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:38.862521    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podStartSLOduration=1.862506212 podStartE2EDuration="1.862506212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:41:38.861296191 +0000 UTC m=+1088.538883137" watchObservedRunningTime="2026-11-01 00:41:38.862506212 +0000 UTC m=+1088.540093157"
Nov 01 00:41:39.112248 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:39.112217    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:44.989000 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:41:44.988968    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:41:45.795711 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.795674    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 00:41:45.798235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.798220    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805004    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2bj\" (UniqueName: \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805183    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805242    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805334    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.805611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.805370    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.901307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.901263    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"simulator-w2-4143b7-kserve-self-signed-certs\""
Nov 01 00:41:45.901307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.901263    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"default-dockercfg-g7cxq\""
Nov 01 00:41:45.901528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.901292    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"openshift-service-ca.crt\""
Nov 01 00:41:45.901528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.901297    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"kube-root-ca.crt\""
Nov 01 00:41:45.904002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.903982    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 00:41:45.906014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.905995    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906104 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906025    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906104 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906053    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2bj\" (UniqueName: \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906116    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906175    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906196    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906495 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906472    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906501    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.906607 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.906589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.908588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.908562    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.908693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.908670    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:45.967735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:45.967707    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2bj\" (UniqueName: \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\") pod \"simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") " pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:46.110665 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:46.110562    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:46.490339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:46.490318    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 00:41:46.492502 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:41:46.492472    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cba285c_f5c9_4a51_8e8c_1bead9e00439.slice/crio-532a29c70c1e9b5424478fb83aaf42b8731fd39567c5b97f71f8706dc5cef7fa WatchSource:0}: Error finding container 532a29c70c1e9b5424478fb83aaf42b8731fd39567c5b97f71f8706dc5cef7fa: Status 404 returned error can't find the container with id 532a29c70c1e9b5424478fb83aaf42b8731fd39567c5b97f71f8706dc5cef7fa
Nov 01 00:41:46.772948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:46.772843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" event={"ID":"8cba285c-f5c9-4a51-8e8c-1bead9e00439","Type":"ContainerStarted","Data":"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"}
Nov 01 00:41:46.772948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:46.772892    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" event={"ID":"8cba285c-f5c9-4a51-8e8c-1bead9e00439","Type":"ContainerStarted","Data":"532a29c70c1e9b5424478fb83aaf42b8731fd39567c5b97f71f8706dc5cef7fa"}
Nov 01 00:41:46.892562 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:46.892502    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podStartSLOduration=1.8924831050000002 podStartE2EDuration="1.892483105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:41:46.89241546 +0000 UTC m=+1096.570002406" watchObservedRunningTime="2026-11-01 00:41:46.892483105 +0000 UTC m=+1096.570070052"
Nov 01 00:41:47.111735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:47.111659    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:48.112288 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:48.112250    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:48.124935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:48.124903    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:48.793337 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:48.793311    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:41:55.897923 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:55.897893    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:41:56.111523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:56.111488    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:56.124430 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:56.124400    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:41:56.845943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:41:56.845908    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:42:14.989506 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:42:14.989432    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:42:26.241669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:26.241635    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:42:38.584544 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.584511    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 00:42:38.588354 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.588338    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.640971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.640945    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"premium-w1-30fb19-kserve-self-signed-certs\""
Nov 01 00:42:38.644322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644290    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.644534 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644360    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.644534 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644520    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.644685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644550    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfbpl\" (UniqueName: \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.644685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644577    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.644799 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.644777    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.655809 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.655789    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"openshift-service-ca.crt\""
Nov 01 00:42:38.690955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.690932    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"default-dockercfg-b2lgb\""
Nov 01 00:42:38.690955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.690942    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"kube-root-ca.crt\""
Nov 01 00:42:38.693304 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.693283    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 00:42:38.745741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745709    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.745887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745783    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.745887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745809    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.745887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745866    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.746020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745888    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfbpl\" (UniqueName: \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.746020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.745912    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.746247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.746220    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.746329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.746279    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.746329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.746319    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.748338 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.748316    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.748496 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.748481    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.808157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.808100    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfbpl\" (UniqueName: \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\") pod \"premium-w1-30fb19-kserve-6c8874ffd6-txsbm\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") " pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:38.899300 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:38.899209    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:39.301293 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:39.301273    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 00:42:39.303753 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:42:39.303721    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60ab262_9c75_487e_9031_7a0fa02e387c.slice/crio-8070b18c9d370b63f4c3793987ba5259d92d8837b8b5ace1c39fb7e8d97f398e WatchSource:0}: Error finding container 8070b18c9d370b63f4c3793987ba5259d92d8837b8b5ace1c39fb7e8d97f398e: Status 404 returned error can't find the container with id 8070b18c9d370b63f4c3793987ba5259d92d8837b8b5ace1c39fb7e8d97f398e
Nov 01 00:42:40.039492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:40.039460    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" event={"ID":"b60ab262-9c75-487e-9031-7a0fa02e387c","Type":"ContainerStarted","Data":"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"}
Nov 01 00:42:40.039877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:40.039502    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" event={"ID":"b60ab262-9c75-487e-9031-7a0fa02e387c","Type":"ContainerStarted","Data":"8070b18c9d370b63f4c3793987ba5259d92d8837b8b5ace1c39fb7e8d97f398e"}
Nov 01 00:42:40.157056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:40.157008    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podStartSLOduration=2.156993199 podStartE2EDuration="2.156993199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:42:40.156497977 +0000 UTC m=+1149.834084923" watchObservedRunningTime="2026-11-01 00:42:40.156993199 +0000 UTC m=+1149.834580143"
Nov 01 00:42:40.899917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:40.899882    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:44.990492 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:42:44.990457    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:42:48.900338 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:48.900294    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:48.912882 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:48.912858    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:49.094004 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:49.093971    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:42:49.808007 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:42:49.807973    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:43:11.312018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:11.311985    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:43:14.988675 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:43:14.988640    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:43:28.637178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.637144    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 00:43:28.640734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.640717    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.693316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.693290    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"openshift-service-ca.crt\""
Nov 01 00:43:28.743266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.743220    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"kube-root-ca.crt\""
Nov 01 00:43:28.743442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.743355    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"default-dockercfg-n9ktj\""
Nov 01 00:43:28.743485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.743472    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"premium-w3-5a9c19-kserve-self-signed-certs\""
Nov 01 00:43:28.745468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.745447    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.786910    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.786974    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.787016    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hj6v\" (UniqueName: \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.787073    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.787106    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.788068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.787175    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888178    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888219    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888251    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888461    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888513    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hj6v\" (UniqueName: \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.888718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888633    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.897210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.888690    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.897210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.889483    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.897210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.891080    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.897210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.896635    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.950538 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.950509    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hj6v\" (UniqueName: \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\") pod \"premium-w3-5a9c19-kserve-6cd969db4c-4gwfb\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") " pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:28.953222 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:28.953207    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:29.333527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:29.333501    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 00:43:29.335629 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:43:29.335603    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4c8c600_6f30_4c6b_bf55_281ae7918492.slice/crio-c357a5206a359e6b2b287c609a12a649fa5f7a6aaba29ef936ddf2fd1c021c39 WatchSource:0}: Error finding container c357a5206a359e6b2b287c609a12a649fa5f7a6aaba29ef936ddf2fd1c021c39: Status 404 returned error can't find the container with id c357a5206a359e6b2b287c609a12a649fa5f7a6aaba29ef936ddf2fd1c021c39
Nov 01 00:43:30.314142 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:30.314089    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" event={"ID":"b4c8c600-6f30-4c6b-bf55-281ae7918492","Type":"ContainerStarted","Data":"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"}
Nov 01 00:43:30.314542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:30.314169    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" event={"ID":"b4c8c600-6f30-4c6b-bf55-281ae7918492","Type":"ContainerStarted","Data":"c357a5206a359e6b2b287c609a12a649fa5f7a6aaba29ef936ddf2fd1c021c39"}
Nov 01 00:43:30.434709 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:30.434646    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podStartSLOduration=2.434628583 podStartE2EDuration="2.434628583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:30.43224783 +0000 UTC m=+1200.109834771" watchObservedRunningTime="2026-11-01 00:43:30.434628583 +0000 UTC m=+1200.112215528"
Nov 01 00:43:30.953726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:30.953531    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:34.704965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.704933    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 00:43:34.708651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.708633    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.750648 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.750616    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.750648 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.750652    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.750972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.750670    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.750972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.750876    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.750972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.750947    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.751152 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.751037    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slz9g\" (UniqueName: \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.811183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.811118    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"openshift-service-ca.crt\""
Nov 01 00:43:34.811183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.811123    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"premium-w0-32439f-kserve-self-signed-certs\""
Nov 01 00:43:34.811183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.811125    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"default-dockercfg-d59kz\""
Nov 01 00:43:34.811427 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.811223    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"kube-root-ca.crt\""
Nov 01 00:43:34.813373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.813356    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 00:43:34.856066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856012    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.856316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856147    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.856387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slz9g\" (UniqueName: \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.856449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.856510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856447    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.856577 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.856514    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.857376 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.857307    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.857767 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.857739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.858233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.858212    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.864546 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.864484    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.865014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.864990    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:34.919046 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:34.919007    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slz9g\" (UniqueName: \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\") pod \"premium-w0-32439f-kserve-674495c4c5-7sc2d\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") " pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:35.031970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:35.031941    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:35.421196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:35.421108    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 00:43:35.424197 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:43:35.424170    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d23648_a719_40e5_873b_a05e69faddad.slice/crio-c818fa1035fd9dc271ec86eb206fd593f9c3cc5068657d766e9d1c6d2eea9bf5 WatchSource:0}: Error finding container c818fa1035fd9dc271ec86eb206fd593f9c3cc5068657d766e9d1c6d2eea9bf5: Status 404 returned error can't find the container with id c818fa1035fd9dc271ec86eb206fd593f9c3cc5068657d766e9d1c6d2eea9bf5
Nov 01 00:43:36.352709 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:36.352676    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" event={"ID":"e8d23648-a719-40e5-873b-a05e69faddad","Type":"ContainerStarted","Data":"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"}
Nov 01 00:43:36.352709 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:36.352715    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" event={"ID":"e8d23648-a719-40e5-873b-a05e69faddad","Type":"ContainerStarted","Data":"c818fa1035fd9dc271ec86eb206fd593f9c3cc5068657d766e9d1c6d2eea9bf5"}
Nov 01 00:43:36.471373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:36.471323    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podStartSLOduration=2.471308885 podStartE2EDuration="2.471308885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:36.469353467 +0000 UTC m=+1206.146940413" watchObservedRunningTime="2026-11-01 00:43:36.471308885 +0000 UTC m=+1206.148895830"
Nov 01 00:43:37.032440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:37.032409    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:38.954194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:38.954161    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:38.966639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:38.966613    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:39.378609 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:39.378580    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:43:43.045979 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:43.045902    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:43:44.990409 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:43:44.990369    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:43:45.032679 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:45.032639    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:45.046240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:45.046213    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:43:45.411501 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:43:45.411424    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:44:08.215219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:08.215184    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:44:08.215731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:08.215416    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy" containerID="cri-o://644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3" gracePeriod=30
Nov 01 00:44:13.474978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.474944    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:44:13.539943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.539862    4493 generic.go:356] "Generic (PLEG): container finished" podID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerID="644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3" exitCode=0
Nov 01 00:44:13.539943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.539903    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"
Nov 01 00:44:13.540148 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.539938    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" event={"ID":"a87ed863-cd59-4d89-8d23-a6be691bff76","Type":"ContainerDied","Data":"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"}
Nov 01 00:44:13.540148 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.539989    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s" event={"ID":"a87ed863-cd59-4d89-8d23-a6be691bff76","Type":"ContainerDied","Data":"77681753903a851fa5abb239e58836c7997278febca2b3cc8fc79a39f83357f7"}
Nov 01 00:44:13.540148 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.540010    4493 scope.go:122] "RemoveContainer" containerID="644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"
Nov 01 00:44:13.549092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.549071    4493 scope.go:122] "RemoveContainer" containerID="644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"
Nov 01 00:44:13.549375 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:44:13.549354    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3\": container with ID starting with 644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3 not found: ID does not exist" containerID="644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"
Nov 01 00:44:13.549453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.549382    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3"} err="failed to get container status \"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3\": rpc error: code = NotFound desc = could not find container \"644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3\": container with ID starting with 644b8756879b1e55c16b56cf98b7d233b6bdcc26bcad366289f494b736a3c6c3 not found: ID does not exist"
Nov 01 00:44:13.577622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577596    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.577782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577629    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.577782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577670    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.577782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577707    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.577782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577726    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.577782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577764    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.578044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577800    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.578044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577876    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.578044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\") pod \"a87ed863-cd59-4d89-8d23-a6be691bff76\" (UID: \"a87ed863-cd59-4d89-8d23-a6be691bff76\") "
Nov 01 00:44:13.578044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.577916    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:44:13.578286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578056    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:44:13.578286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578105    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:44:13.578401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578291    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:44:13.578754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578599    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.578754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578716    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.578754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578739    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87ed863-cd59-4d89-8d23-a6be691bff76-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.578971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578756    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.578971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.578841    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:44:13.580355 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.580327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:44:13.580355 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.580333    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:44:13.580636 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.580611    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:44:13.580724 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.580674    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw" pod "a87ed863-cd59-4d89-8d23-a6be691bff76" (UID: "a87ed863-cd59-4d89-8d23-a6be691bff76"). InnerVolumeSpecName "kube-api-access-fk2bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:44:13.680080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.680045    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.680080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.680076    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.680080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.680086    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.680386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.680095    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2bw\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-kube-api-access-fk2bw\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.680386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.680106    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87ed863-cd59-4d89-8d23-a6be691bff76-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:44:13.963542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.963505    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:44:13.969757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:13.969724    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-86c2s"]
Nov 01 00:44:14.503994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:14.503961    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" path="/var/lib/kubelet/pods/a87ed863-cd59-4d89-8d23-a6be691bff76/volumes"
Nov 01 00:44:14.999917 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:44:14.999878    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:44:21.357923 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:21.357889    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:44:31.785194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.785162    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:44:31.785626 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.785614    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy"
Nov 01 00:44:31.785669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.785628    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy"
Nov 01 00:44:31.785723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.785715    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87ed863-cd59-4d89-8d23-a6be691bff76" containerName="istio-proxy"
Nov 01 00:44:31.789938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.789921    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.846571 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.846542    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-948e22-openshift-default-dockercfg-2wgzk\""
Nov 01 00:44:31.899010 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.898982    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:44:31.901124 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901098    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901167    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901285    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782dq\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901499 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901482    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901532 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901515    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901552    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:31.901661 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:31.901640    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003216 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003173    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003251    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003283    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782dq\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003315    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003345    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003379    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003429    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003483    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003625    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003670    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.003991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003972    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.004071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.003992    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.004071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.004012    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.005831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.005810    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.006147 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.006111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.065207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.065104    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.065374 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.065267    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782dq\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.103267 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.103233    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:32.535042 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.535020    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:44:32.536950 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:44:32.536920    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26aaa66_5506_4b70_b8ad_b2e77d633aeb.slice/crio-7492eedcb4b303a0f8cd65a845d3b607c670d2ccd9872e92f1c42243f6a6db24 WatchSource:0}: Error finding container 7492eedcb4b303a0f8cd65a845d3b607c670d2ccd9872e92f1c42243f6a6db24: Status 404 returned error can't find the container with id 7492eedcb4b303a0f8cd65a845d3b607c670d2ccd9872e92f1c42243f6a6db24
Nov 01 00:44:32.542255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.542222    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:44:32.542346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.542290    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:44:32.542346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.542316    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:44:32.683743 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.683705    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" event={"ID":"e26aaa66-5506-4b70-b8ad-b2e77d633aeb","Type":"ContainerStarted","Data":"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"}
Nov 01 00:44:32.683909 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.683759    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" event={"ID":"e26aaa66-5506-4b70-b8ad-b2e77d633aeb","Type":"ContainerStarted","Data":"7492eedcb4b303a0f8cd65a845d3b607c670d2ccd9872e92f1c42243f6a6db24"}
Nov 01 00:44:32.804793 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:32.804674    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podStartSLOduration=1.804654449 podStartE2EDuration="1.804654449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:32.803763476 +0000 UTC m=+1262.481350425" watchObservedRunningTime="2026-11-01 00:44:32.804654449 +0000 UTC m=+1262.482241396"
Nov 01 00:44:33.103516 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:33.103435    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:33.105121 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:33.105093    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:33.105294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:33.105170    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:34.104388 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:34.104353    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:34.104803 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:34.104410    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:35.103716 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:35.103683    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:35.103897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:35.103744    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:36.104017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:36.103978    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:36.104477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:36.104044    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:37.104048 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:37.104010    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:37.104501 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:37.104063    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:38.103768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:38.103729    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 00:44:38.103949 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:38.103804    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 00:44:39.244494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:39.244449    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:39.245685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:39.245660    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:39.246470 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:44:39.246449    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:44:44.989802 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:44:44.989753    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:45:00.174821 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.174790    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:00.178419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.178400    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.193104 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.193084    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:00.196413 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.196399    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.241009 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.240936    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 00:45:00.282347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.282314    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:00.282860 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.282836    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.282985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.282924    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpls\" (UniqueName: \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.282985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.282961    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.283101 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.283001    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv45\" (UniqueName: \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.347389 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.347348    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:00.383794 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.383755    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.383955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.383804    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv45\" (UniqueName: \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.383955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.383929    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.384065 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.384020    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpls\" (UniqueName: \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.384504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.384485    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.384550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.384508    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.448715 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.448639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpls\" (UniqueName: \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.448715 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.448639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv45\" (UniqueName: \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.492173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.492115    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:00.508542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.508518    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:00.870796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.870772    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:00.872925 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:00.872899    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a472525_3b64_49d6_84b8_93b2ee39fd2d.slice/crio-b0c26cd6ad76b98b6d82a8582e852cd7b8b57beb7bda5785bec0bf46f236c0d4 WatchSource:0}: Error finding container b0c26cd6ad76b98b6d82a8582e852cd7b8b57beb7bda5785bec0bf46f236c0d4: Status 404 returned error can't find the container with id b0c26cd6ad76b98b6d82a8582e852cd7b8b57beb7bda5785bec0bf46f236c0d4
Nov 01 00:45:00.874763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.874744    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:45:00.919417 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:00.919392    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:00.924477 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:00.924448    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c7b49a_e386_404b_b81a_2f12a87cd32f.slice/crio-766038660b54567b8740afe276ba0523f74fb998d06ea6f1f0871f20211b736f WatchSource:0}: Error finding container 766038660b54567b8740afe276ba0523f74fb998d06ea6f1f0871f20211b736f: Status 404 returned error can't find the container with id 766038660b54567b8740afe276ba0523f74fb998d06ea6f1f0871f20211b736f
Nov 01 00:45:01.137910 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:01.137825    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:45:01.880163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:01.880089    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerStarted","Data":"766038660b54567b8740afe276ba0523f74fb998d06ea6f1f0871f20211b736f"}
Nov 01 00:45:01.881597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:01.881574    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerStarted","Data":"b0c26cd6ad76b98b6d82a8582e852cd7b8b57beb7bda5785bec0bf46f236c0d4"}
Nov 01 00:45:05.909525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.909490    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerID="5f05d10d001be47c80647bbda0f6d3a7b3dca7fee87b6daf9d0fed685ea50c63" exitCode=60
Nov 01 00:45:05.909958 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.909576    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerDied","Data":"5f05d10d001be47c80647bbda0f6d3a7b3dca7fee87b6daf9d0fed685ea50c63"}
Nov 01 00:45:05.909958 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.909835    4493 scope.go:122] "RemoveContainer" containerID="5f05d10d001be47c80647bbda0f6d3a7b3dca7fee87b6daf9d0fed685ea50c63"
Nov 01 00:45:05.911313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.911287    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerID="1bde6cc4dab0e55614c8a9e980ddbb5aeff811d6cb5999faf753efcf2ff89592" exitCode=60
Nov 01 00:45:05.911397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.911347    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerDied","Data":"1bde6cc4dab0e55614c8a9e980ddbb5aeff811d6cb5999faf753efcf2ff89592"}
Nov 01 00:45:05.911593 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:05.911579    4493 scope.go:122] "RemoveContainer" containerID="1bde6cc4dab0e55614c8a9e980ddbb5aeff811d6cb5999faf753efcf2ff89592"
Nov 01 00:45:06.922729 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.922696    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerID="6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2" exitCode=60
Nov 01 00:45:06.923257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.922789    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerDied","Data":"6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2"}
Nov 01 00:45:06.923257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.922849    4493 scope.go:122] "RemoveContainer" containerID="5f05d10d001be47c80647bbda0f6d3a7b3dca7fee87b6daf9d0fed685ea50c63"
Nov 01 00:45:06.923257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.923181    4493 scope.go:122] "RemoveContainer" containerID="6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2"
Nov 01 00:45:06.923523 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:06.923500    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn_odh-ai-gateway-infra(a3c7b49a-e386-404b-b81a-2f12a87cd32f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f"
Nov 01 00:45:06.924967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.924948    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerID="2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939" exitCode=60
Nov 01 00:45:06.925050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.925026    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerDied","Data":"2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939"}
Nov 01 00:45:06.925321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.925300    4493 scope.go:122] "RemoveContainer" containerID="2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939"
Nov 01 00:45:06.925547 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:06.925522    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz_odh-ai-gateway-infra(5a472525-3b64-49d6-84b8-93b2ee39fd2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d"
Nov 01 00:45:06.940844 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:06.940825    4493 scope.go:122] "RemoveContainer" containerID="1bde6cc4dab0e55614c8a9e980ddbb5aeff811d6cb5999faf753efcf2ff89592"
Nov 01 00:45:07.932173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:07.932150    4493 scope.go:122] "RemoveContainer" containerID="2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939"
Nov 01 00:45:07.932620 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:07.932425    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz_odh-ai-gateway-infra(5a472525-3b64-49d6-84b8-93b2ee39fd2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d"
Nov 01 00:45:07.933738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:07.933720    4493 scope.go:122] "RemoveContainer" containerID="6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2"
Nov 01 00:45:07.933937 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:07.933920    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn_odh-ai-gateway-infra(a3c7b49a-e386-404b-b81a-2f12a87cd32f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f"
Nov 01 00:45:08.259124 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.259090    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"]
Nov 01 00:45:08.263865 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.263848    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.284897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.284871    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.285064 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.284927    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.285064 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.285032    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8k8\" (UniqueName: \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.320107 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.319956    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 00:45:08.324157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.323525    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"]
Nov 01 00:45:08.386391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.386359    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.386585 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.386441    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.386585 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.386471    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8k8\" (UniqueName: \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.387098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.387075    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.389025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.389004    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.447787 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.447759    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8k8\" (UniqueName: \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\") pod \"maas-api-revoke-sub-64579a35-sb2f5\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.577098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:08.577005    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:08.997806 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:08.997757    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfefe2ae_a027_4f21_b742_81c2caa58e21.slice/crio-d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d WatchSource:0}: Error finding container d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d: Status 404 returned error can't find the container with id d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d
Nov 01 00:45:09.001870 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.001576    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"]
Nov 01 00:45:09.456564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.456525    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"]
Nov 01 00:45:09.459925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.459902    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.508550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.508500    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.508550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.508547    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.508907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.508883    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cm2d\" (UniqueName: \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.519253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.519162    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"]
Nov 01 00:45:09.609740 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.609651    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.609740 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.609688    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.609925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.609758    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cm2d\" (UniqueName: \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.610375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.610356    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.612335 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.612305    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.675230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.675199    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cm2d\" (UniqueName: \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\") pod \"maas-api-revoke-sub-f4c6ce84-qnz4m\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.773910 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.773878    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:09.983018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.982983    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5" event={"ID":"bfefe2ae-a027-4f21-b742-81c2caa58e21","Type":"ContainerStarted","Data":"be0f4183f151ad91eb68f9b327bb45b570c1e761359b3de28226009744527fbc"}
Nov 01 00:45:09.983215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:09.983041    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5" event={"ID":"bfefe2ae-a027-4f21-b742-81c2caa58e21","Type":"ContainerStarted","Data":"d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d"}
Nov 01 00:45:10.099461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.099405    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5" podStartSLOduration=2.099386061 podStartE2EDuration="2.099386061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:10.098047137 +0000 UTC m=+1299.775634077" watchObservedRunningTime="2026-11-01 00:45:10.099386061 +0000 UTC m=+1299.776973009"
Nov 01 00:45:10.151565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.151534    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"]
Nov 01 00:45:10.153464 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:10.153424    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5c344c_a353_4f6b_87c0_254a527f53f5.slice/crio-f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606 WatchSource:0}: Error finding container f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606: Status 404 returned error can't find the container with id f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606
Nov 01 00:45:10.990464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.990430    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m" event={"ID":"ee5c344c-a353-4f6b-87c0-254a527f53f5","Type":"ContainerStarted","Data":"305b34efac5a44bb5e68fc252abfc9121968bd503454472db4083b8cb68adf86"}
Nov 01 00:45:10.990641 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.990492    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m" event={"ID":"ee5c344c-a353-4f6b-87c0-254a527f53f5","Type":"ContainerStarted","Data":"f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606"}
Nov 01 00:45:10.992080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.992058    4493 generic.go:356] "Generic (PLEG): container finished" podID="bfefe2ae-a027-4f21-b742-81c2caa58e21" containerID="be0f4183f151ad91eb68f9b327bb45b570c1e761359b3de28226009744527fbc" exitCode=0
Nov 01 00:45:10.992224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:10.992164    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5" event={"ID":"bfefe2ae-a027-4f21-b742-81c2caa58e21","Type":"ContainerDied","Data":"be0f4183f151ad91eb68f9b327bb45b570c1e761359b3de28226009744527fbc"}
Nov 01 00:45:11.107070 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:11.107000    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m" podStartSLOduration=2.106981803 podStartE2EDuration="2.106981803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:11.106010501 +0000 UTC m=+1300.783597447" watchObservedRunningTime="2026-11-01 00:45:11.106981803 +0000 UTC m=+1300.784568749"
Nov 01 00:45:11.999220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:11.999115    4493 generic.go:356] "Generic (PLEG): container finished" podID="ee5c344c-a353-4f6b-87c0-254a527f53f5" containerID="305b34efac5a44bb5e68fc252abfc9121968bd503454472db4083b8cb68adf86" exitCode=0
Nov 01 00:45:11.999220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:11.999202    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m" event={"ID":"ee5c344c-a353-4f6b-87c0-254a527f53f5","Type":"ContainerDied","Data":"305b34efac5a44bb5e68fc252abfc9121968bd503454472db4083b8cb68adf86"}
Nov 01 00:45:12.144954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.144927    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:12.237941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.237898    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\" (UniqueName: \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\") pod \"bfefe2ae-a027-4f21-b742-81c2caa58e21\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") "
Nov 01 00:45:12.238162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.238009    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\") pod \"bfefe2ae-a027-4f21-b742-81c2caa58e21\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") "
Nov 01 00:45:12.238162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.238092    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\") pod \"bfefe2ae-a027-4f21-b742-81c2caa58e21\" (UID: \"bfefe2ae-a027-4f21-b742-81c2caa58e21\") "
Nov 01 00:45:12.238400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.238377    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca" pod "bfefe2ae-a027-4f21-b742-81c2caa58e21" (UID: "bfefe2ae-a027-4f21-b742-81c2caa58e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:45:12.238472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.238456    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:12.240290 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.240263    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca" pod "bfefe2ae-a027-4f21-b742-81c2caa58e21" (UID: "bfefe2ae-a027-4f21-b742-81c2caa58e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:45:12.240386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.240327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8" pod "bfefe2ae-a027-4f21-b742-81c2caa58e21" (UID: "bfefe2ae-a027-4f21-b742-81c2caa58e21"). InnerVolumeSpecName "kube-api-access-wh8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:45:12.339253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.339127    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefe2ae-a027-4f21-b742-81c2caa58e21-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:12.339253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:12.339191    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8k8\" (UniqueName: \"kubernetes.io/projected/bfefe2ae-a027-4f21-b742-81c2caa58e21-kube-api-access-wh8k8\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:13.006298 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.006270    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"
Nov 01 00:45:13.006469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.006292    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5" event={"ID":"bfefe2ae-a027-4f21-b742-81c2caa58e21","Type":"ContainerDied","Data":"d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d"}
Nov 01 00:45:13.006469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.006389    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4875891102e83edba53925a44ba923ef90f59e93288ec3f81d993b7e2e9345d"
Nov 01 00:45:13.175907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.175881    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:13.250431 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.250393    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\") pod \"ee5c344c-a353-4f6b-87c0-254a527f53f5\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") "
Nov 01 00:45:13.250634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.250455    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\") pod \"ee5c344c-a353-4f6b-87c0-254a527f53f5\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") "
Nov 01 00:45:13.250634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.250514    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\" (UniqueName: \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\") pod \"ee5c344c-a353-4f6b-87c0-254a527f53f5\" (UID: \"ee5c344c-a353-4f6b-87c0-254a527f53f5\") "
Nov 01 00:45:13.250807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.250783    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca" pod "ee5c344c-a353-4f6b-87c0-254a527f53f5" (UID: "ee5c344c-a353-4f6b-87c0-254a527f53f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:45:13.252762 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.252738    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca" pod "ee5c344c-a353-4f6b-87c0-254a527f53f5" (UID: "ee5c344c-a353-4f6b-87c0-254a527f53f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:45:13.252874 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.252818    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d" pod "ee5c344c-a353-4f6b-87c0-254a527f53f5" (UID: "ee5c344c-a353-4f6b-87c0-254a527f53f5"). InnerVolumeSpecName "kube-api-access-2cm2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:45:13.351364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.351323    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:13.351364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.351358    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5c344c-a353-4f6b-87c0-254a527f53f5-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:13.351364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.351372    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cm2d\" (UniqueName: \"kubernetes.io/projected/ee5c344c-a353-4f6b-87c0-254a527f53f5-kube-api-access-2cm2d\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:13.503487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.503435    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"]
Nov 01 00:45:13.504196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504173    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bfefe2ae-a027-4f21-b742-81c2caa58e21" containerName="revoke-keys"
Nov 01 00:45:13.504283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504199    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefe2ae-a027-4f21-b742-81c2caa58e21" containerName="revoke-keys"
Nov 01 00:45:13.504283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504217    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ee5c344c-a353-4f6b-87c0-254a527f53f5" containerName="revoke-keys"
Nov 01 00:45:13.504283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504225    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5c344c-a353-4f6b-87c0-254a527f53f5" containerName="revoke-keys"
Nov 01 00:45:13.504393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504380    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefe2ae-a027-4f21-b742-81c2caa58e21" containerName="revoke-keys"
Nov 01 00:45:13.504429 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.504396    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5c344c-a353-4f6b-87c0-254a527f53f5" containerName="revoke-keys"
Nov 01 00:45:13.507919 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.507898    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.552616 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.552581    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.552786 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.552687    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.552786 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.552729    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdkd\" (UniqueName: \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.565969 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.565941    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"]
Nov 01 00:45:13.653579 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.653525    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.653754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.653605    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.653754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.653627    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdkd\" (UniqueName: \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.654319 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.654256    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.656013 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.655988    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.714908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.714875    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdkd\" (UniqueName: \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\") pod \"maas-api-revoke-sub-a00cda00-dshzm\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:13.821381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:13.821298    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:14.013024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:14.012990    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"
Nov 01 00:45:14.013235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:14.012993    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m" event={"ID":"ee5c344c-a353-4f6b-87c0-254a527f53f5","Type":"ContainerDied","Data":"f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606"}
Nov 01 00:45:14.013235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:14.013123    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f713824624f59492216a776ae22e04c361963a720f854b1e13451c8265f4c606"
Nov 01 00:45:14.197914 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:14.197884    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"]
Nov 01 00:45:14.199538 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:14.199508    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224c039b_fad5_48f9_842f_c63072db840b.slice/crio-56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105 WatchSource:0}: Error finding container 56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105: Status 404 returned error can't find the container with id 56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105
Nov 01 00:45:14.988998 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:14.988966    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:45:15.019631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:15.019596    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm" event={"ID":"224c039b-fad5-48f9-842f-c63072db840b","Type":"ContainerStarted","Data":"8cdff806c44e4cff8e0639a62da8505476450e27ce894dc9dd8a1f549b36cf5b"}
Nov 01 00:45:15.019786 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:15.019647    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm" event={"ID":"224c039b-fad5-48f9-842f-c63072db840b","Type":"ContainerStarted","Data":"56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105"}
Nov 01 00:45:15.144421 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:15.144376    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm" podStartSLOduration=2.144361145 podStartE2EDuration="2.144361145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:15.141961588 +0000 UTC m=+1304.819548533" watchObservedRunningTime="2026-11-01 00:45:15.144361145 +0000 UTC m=+1304.821948090"
Nov 01 00:45:16.029835 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:16.029800    4493 generic.go:356] "Generic (PLEG): container finished" podID="224c039b-fad5-48f9-842f-c63072db840b" containerID="8cdff806c44e4cff8e0639a62da8505476450e27ce894dc9dd8a1f549b36cf5b" exitCode=0
Nov 01 00:45:16.030370 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:16.029886    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm" event={"ID":"224c039b-fad5-48f9-842f-c63072db840b","Type":"ContainerDied","Data":"8cdff806c44e4cff8e0639a62da8505476450e27ce894dc9dd8a1f549b36cf5b"}
Nov 01 00:45:17.480040 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.480004    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:17.549221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.549183    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\" (UniqueName: \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\") pod \"224c039b-fad5-48f9-842f-c63072db840b\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") "
Nov 01 00:45:17.549549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.549526    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\") pod \"224c039b-fad5-48f9-842f-c63072db840b\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") "
Nov 01 00:45:17.549804 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.549783    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\") pod \"224c039b-fad5-48f9-842f-c63072db840b\" (UID: \"224c039b-fad5-48f9-842f-c63072db840b\") "
Nov 01 00:45:17.551002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.550974    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca" pod "224c039b-fad5-48f9-842f-c63072db840b" (UID: "224c039b-fad5-48f9-842f-c63072db840b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:45:17.552256 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.551718    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd" pod "224c039b-fad5-48f9-842f-c63072db840b" (UID: "224c039b-fad5-48f9-842f-c63072db840b"). InnerVolumeSpecName "kube-api-access-mcdkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:45:17.552622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.552595    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca" pod "224c039b-fad5-48f9-842f-c63072db840b" (UID: "224c039b-fad5-48f9-842f-c63072db840b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:45:17.654240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.654117    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224c039b-fad5-48f9-842f-c63072db840b-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:17.654240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.654185    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdkd\" (UniqueName: \"kubernetes.io/projected/224c039b-fad5-48f9-842f-c63072db840b-kube-api-access-mcdkd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:17.654240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:17.654207    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224c039b-fad5-48f9-842f-c63072db840b-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:18.044807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:18.044775    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"
Nov 01 00:45:18.044807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:18.044805    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm" event={"ID":"224c039b-fad5-48f9-842f-c63072db840b","Type":"ContainerDied","Data":"56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105"}
Nov 01 00:45:18.045034 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:18.044853    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56df8c4d2ebde577fc107ba0ae149f7470f2c95eab263d88e89c9e3992d67105"
Nov 01 00:45:19.496033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:19.495996    4493 scope.go:122] "RemoveContainer" containerID="6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2"
Nov 01 00:45:20.060647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:20.060610    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerStarted","Data":"2c3bb957e88e0688481da0e88660290a3d94cff9ba55780907264b7a881420e8"}
Nov 01 00:45:20.178368 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:20.178262    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" podStartSLOduration=16.398374689 podStartE2EDuration="20.178248171s" totalImagesPullingTime="3.779873482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:20.175602666 +0000 UTC m=+1309.853189609" watchObservedRunningTime="2026-11-01 00:45:20.178248171 +0000 UTC m=+1309.855835115"
Nov 01 00:45:20.502459 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:20.502433    4493 scope.go:122] "RemoveContainer" containerID="2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939"
Nov 01 00:45:21.068357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.068328    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerID="2c3bb957e88e0688481da0e88660290a3d94cff9ba55780907264b7a881420e8" exitCode=60
Nov 01 00:45:21.068552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.068410    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerDied","Data":"2c3bb957e88e0688481da0e88660290a3d94cff9ba55780907264b7a881420e8"}
Nov 01 00:45:21.068552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.068507    4493 scope.go:122] "RemoveContainer" containerID="6462050f5250281da5e49427610dd11549aa1322a2e1c2dd04b05392fb823ef2"
Nov 01 00:45:21.068883 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.068860    4493 scope.go:122] "RemoveContainer" containerID="2c3bb957e88e0688481da0e88660290a3d94cff9ba55780907264b7a881420e8"
Nov 01 00:45:21.069380 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:21.069353    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn_odh-ai-gateway-infra(a3c7b49a-e386-404b-b81a-2f12a87cd32f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f"
Nov 01 00:45:21.070405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.070386    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerStarted","Data":"b11dddb1e52d630e5cd0598c50cb1781e0f1fac12366ee930137810a7dff9af3"}
Nov 01 00:45:21.185907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.185875    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:21.313194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:21.313126    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" podStartSLOduration=17.362014367 podStartE2EDuration="21.313108084s" totalImagesPullingTime="3.951093717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:21.312591235 +0000 UTC m=+1310.990178192" watchObservedRunningTime="2026-11-01 00:45:21.313108084 +0000 UTC m=+1310.990695030"
Nov 01 00:45:22.079857 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.079828    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerID="b11dddb1e52d630e5cd0598c50cb1781e0f1fac12366ee930137810a7dff9af3" exitCode=60
Nov 01 00:45:22.080259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.079865    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerDied","Data":"b11dddb1e52d630e5cd0598c50cb1781e0f1fac12366ee930137810a7dff9af3"}
Nov 01 00:45:22.080259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.079912    4493 scope.go:122] "RemoveContainer" containerID="2db720068ac11febb072a3e8fd37363b53951fffc54a99490100614b54d62939"
Nov 01 00:45:22.080353 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.080283    4493 scope.go:122] "RemoveContainer" containerID="b11dddb1e52d630e5cd0598c50cb1781e0f1fac12366ee930137810a7dff9af3"
Nov 01 00:45:22.080571 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:22.080552    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz_odh-ai-gateway-infra(5a472525-3b64-49d6-84b8-93b2ee39fd2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d"
Nov 01 00:45:22.224688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.224661    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:22.304179 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.304115    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\" (UniqueName: \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\") pod \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") "
Nov 01 00:45:22.304370 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.304240    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\") pod \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\" (UID: \"a3c7b49a-e386-404b-b81a-2f12a87cd32f\") "
Nov 01 00:45:22.304575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.304552    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca" pod "a3c7b49a-e386-404b-b81a-2f12a87cd32f" (UID: "a3c7b49a-e386-404b-b81a-2f12a87cd32f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:45:22.304643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.304631    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c7b49a-e386-404b-b81a-2f12a87cd32f-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:22.306419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.306392    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45" pod "a3c7b49a-e386-404b-b81a-2f12a87cd32f" (UID: "a3c7b49a-e386-404b-b81a-2f12a87cd32f"). InnerVolumeSpecName "kube-api-access-7gv45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:45:22.326985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.326956    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:22.405533 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:22.405450    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gv45\" (UniqueName: \"kubernetes.io/projected/a3c7b49a-e386-404b-b81a-2f12a87cd32f-kube-api-access-7gv45\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:23.086464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.086435    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"
Nov 01 00:45:23.086464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.086455    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn" event={"ID":"a3c7b49a-e386-404b-b81a-2f12a87cd32f","Type":"ContainerDied","Data":"766038660b54567b8740afe276ba0523f74fb998d06ea6f1f0871f20211b736f"}
Nov 01 00:45:23.087035 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.086520    4493 scope.go:122] "RemoveContainer" containerID="2c3bb957e88e0688481da0e88660290a3d94cff9ba55780907264b7a881420e8"
Nov 01 00:45:23.233318 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.233296    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:23.319995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.319963    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\") pod \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") "
Nov 01 00:45:23.320181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.320104    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\" (UniqueName: \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\") pod \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\" (UID: \"5a472525-3b64-49d6-84b8-93b2ee39fd2d\") "
Nov 01 00:45:23.320349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.320327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca" pod "5a472525-3b64-49d6-84b8-93b2ee39fd2d" (UID: "5a472525-3b64-49d6-84b8-93b2ee39fd2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:45:23.320439 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.320425    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a472525-3b64-49d6-84b8-93b2ee39fd2d-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:23.322331 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.322306    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls" pod "5a472525-3b64-49d6-84b8-93b2ee39fd2d" (UID: "5a472525-3b64-49d6-84b8-93b2ee39fd2d"). InnerVolumeSpecName "kube-api-access-xqpls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:45:23.369196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.369114    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:23.376592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.376570    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891565-p9fqn"]
Nov 01 00:45:23.421385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:23.421362    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqpls\" (UniqueName: \"kubernetes.io/projected/5a472525-3b64-49d6-84b8-93b2ee39fd2d-kube-api-access-xqpls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:45:24.097715 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.097685    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"
Nov 01 00:45:24.098213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.097716    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz" event={"ID":"5a472525-3b64-49d6-84b8-93b2ee39fd2d","Type":"ContainerDied","Data":"b0c26cd6ad76b98b6d82a8582e852cd7b8b57beb7bda5785bec0bf46f236c0d4"}
Nov 01 00:45:24.098213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.097773    4493 scope.go:122] "RemoveContainer" containerID="b11dddb1e52d630e5cd0598c50cb1781e0f1fac12366ee930137810a7dff9af3"
Nov 01 00:45:24.202222 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202188    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 00:45:24.202798 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202775    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202799    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202817    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202824    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202841    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202849    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202861    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202868    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202878    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.202891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202884    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202903    4493 cpu_manager.go:414] "containerMap: removing container" podUID="224c039b-fad5-48f9-842f-c63072db840b" containerName="revoke-keys"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.202910    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224c039b-fad5-48f9-842f-c63072db840b" containerName="revoke-keys"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203022    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203036    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="224c039b-fad5-48f9-842f-c63072db840b" containerName="revoke-keys"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203051    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203060    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203070    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" containerName="cleanup"
Nov 01 00:45:24.203404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.203079    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:24.207624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.207604    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.231867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.231840    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 00:45:24.261345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.261314    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"distinct-w4-43a0cf-kserve-self-signed-certs\""
Nov 01 00:45:24.262283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.262259    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:24.265899 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.265870    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891565-b6vwz"]
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331482    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331544    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331570    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331604    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331641    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.332460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.331667    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2qj\" (UniqueName: \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432566    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432612    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432629    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432650    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432782    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.432916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.432829    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2qj\" (UniqueName: \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.433084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.433065    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.433164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.433076    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.433242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.433177    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.435313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.435288    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.435476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.435458    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.498248 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.498218    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2qj\" (UniqueName: \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\") pod \"distinct-w4-43a0cf-kserve-699b64d9dd-ng29h\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.502063 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.502035    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a472525-3b64-49d6-84b8-93b2ee39fd2d" path="/var/lib/kubelet/pods/5a472525-3b64-49d6-84b8-93b2ee39fd2d/volumes"
Nov 01 00:45:24.502529 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.502509    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" path="/var/lib/kubelet/pods/a3c7b49a-e386-404b-b81a-2f12a87cd32f/volumes"
Nov 01 00:45:24.519794 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.519762    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:24.935237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:24.935210    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 00:45:24.937557 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:24.937532    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb68e8e_7b6e_42b3_adfd_32b0d7694c85.slice/crio-343ae1dca90f0a8af4cffc7a994b2cd0b7f516ca222757c9825f121be3b96aad WatchSource:0}: Error finding container 343ae1dca90f0a8af4cffc7a994b2cd0b7f516ca222757c9825f121be3b96aad: Status 404 returned error can't find the container with id 343ae1dca90f0a8af4cffc7a994b2cd0b7f516ca222757c9825f121be3b96aad
Nov 01 00:45:25.104500 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:25.104464    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" event={"ID":"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85","Type":"ContainerStarted","Data":"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"}
Nov 01 00:45:25.104892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:25.104513    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" event={"ID":"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85","Type":"ContainerStarted","Data":"343ae1dca90f0a8af4cffc7a994b2cd0b7f516ca222757c9825f121be3b96aad"}
Nov 01 00:45:25.224294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:25.224244    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podStartSLOduration=1.224228846 podStartE2EDuration="1.224228846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:25.221663772 +0000 UTC m=+1314.899250719" watchObservedRunningTime="2026-11-01 00:45:25.224228846 +0000 UTC m=+1314.901815791"
Nov 01 00:45:25.520412 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:25.520337    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:34.520429 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:34.520393    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:34.533439 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:34.533418    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:35.176928 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:35.176897    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:45:38.487173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.487119    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:45:38.953210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.952051    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:45:38.953210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.952770    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:38.953210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.952790    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:38.953210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.952941    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c7b49a-e386-404b-b81a-2f12a87cd32f" containerName="cleanup"
Nov 01 00:45:38.962157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:38.961819    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.009979    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010117    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010210    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d9x\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010245    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010515 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010371    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010515 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010422    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010515 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010482    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010531    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.010664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.010571    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.124775 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.113862    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179308    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179620 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.177793    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:45:39.179718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179597    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179722    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179758    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179898 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179879    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.179953 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179924    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d9x\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.180005 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.179970    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.180063 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.180036    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.180559 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.180530    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.181939 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.181389    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.182118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.181618    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.182506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.182486    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.183221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.183196    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.185707 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.185653    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.196010 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.195984    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.200385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.200355    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.244110 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.244075    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d9x\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\") pod \"maas-default-gateway-openshift-default-67bf986f99-cs447\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.300055 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.299990    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:39.790609 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.790561    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:45:39.803145 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.803096    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:39.803254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.803207    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:39.803320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:39.803263    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:40.210091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.209894    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" event={"ID":"10b3d870-d526-42eb-bcd7-fab222214a6c","Type":"ContainerStarted","Data":"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"}
Nov 01 00:45:40.210091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.209961    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" event={"ID":"10b3d870-d526-42eb-bcd7-fab222214a6c","Type":"ContainerStarted","Data":"dc55e689eb35171f2eb2ef3b25790d8094dd311d4d047ac72872f4945faea537"}
Nov 01 00:45:40.301006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.300955    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:40.302589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.302561    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:40.302739 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.302610    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:40.333183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:40.331531    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podStartSLOduration=2.331512183 podStartE2EDuration="2.331512183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:40.331213083 +0000 UTC m=+1330.008800028" watchObservedRunningTime="2026-11-01 00:45:40.331512183 +0000 UTC m=+1330.009099129"
Nov 01 00:45:41.300577 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:41.300545    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:41.300947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:41.300596    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:42.301254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:42.301213    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:42.301627 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:42.301300    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:43.300862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:43.300829    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:43.301042 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:43.300893    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:44.301275 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:44.301241    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:44.301682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:44.301301    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:44.993038 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:45:44.993005    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:45:45.300472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:45.300390    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:45.300472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:45.300442    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:46.300731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:46.300698    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:46.301159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:46.300748    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:47.300948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:47.300909    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cs447 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 00:45:47.301347 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:47.300997    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 00:45:48.396734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:48.396698    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:48.397638 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:48.397619    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:48.398506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:48.398485    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:45:54.056563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.056533    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 00:45:54.060342 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.060325    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.165871 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.165837    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 00:45:54.188461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188459    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzdt\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188497    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188590    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188622    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188913 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188743    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188913 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188779    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.188913 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188819    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.189092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.188955    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290031 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.289998    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzdt\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290047    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290105    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290164    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290249    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290536 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290283    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290536 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290338    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290716 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.290776 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.290758    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.291453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.291074    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.291961 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.291875    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.292205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.292179    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.292340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.292325    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.292453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.292388    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.294956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.294927    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.295046 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.294951    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.351938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.351865    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.352071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.351967    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzdt\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\") pod \"maas-default-gateway-openshift-default-67bf986f99-qfr9d\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.373997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.373968    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:54.806552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.806529    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 00:45:54.807967 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:45:54.807945    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed7b9c2_8011_414b_8bc4_f8acd2bf7421.slice/crio-325598a3d044da0363f78557254998be10c754a6c3fe1c295f686e639f1b776f WatchSource:0}: Error finding container 325598a3d044da0363f78557254998be10c754a6c3fe1c295f686e639f1b776f: Status 404 returned error can't find the container with id 325598a3d044da0363f78557254998be10c754a6c3fe1c295f686e639f1b776f
Nov 01 00:45:54.810356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.810326    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:54.810462 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.810387    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:54.810462 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:54.810416    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:45:55.352693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.352650    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" event={"ID":"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421","Type":"ContainerStarted","Data":"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"}
Nov 01 00:45:55.353170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.352740    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" event={"ID":"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421","Type":"ContainerStarted","Data":"325598a3d044da0363f78557254998be10c754a6c3fe1c295f686e639f1b776f"}
Nov 01 00:45:55.374290 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.374263    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:45:55.376002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.375978    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:45:55.376114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.376023    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:45:55.476932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:55.476868    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podStartSLOduration=1.476848514 podStartE2EDuration="1.476848514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:55.475630482 +0000 UTC m=+1345.153217427" watchObservedRunningTime="2026-11-01 00:45:55.476848514 +0000 UTC m=+1345.154435461"
Nov 01 00:45:56.375366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:56.375324    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:45:56.375753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:56.375382    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:45:57.375592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:57.374650    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:45:57.375592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:57.374720    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:45:58.375253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:58.375215    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:45:58.375427 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:58.375275    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:45:59.375210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:59.375176    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:45:59.375606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:45:59.375256    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:46:00.392158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:00.389082    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:46:00.392158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:00.389161    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:46:01.375159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.374578    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:46:01.375159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.374646    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:46:01.811011 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.810974    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 00:46:01.814576 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.814559    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900175    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900231    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900295    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900321    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900388    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900598 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900424    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900598 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900504    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900598 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8dr\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:01.900718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:01.900651    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001437    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001483    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001499    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001533    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001649    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001733    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001843    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001895    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8dr\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.001960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.001934    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.002203 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.002116    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.002338 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.002308    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.002504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.002395    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.002504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.002402    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.004149 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.004116    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.004542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.004524    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.027606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.027576    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 00:46:02.063734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.063634    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8dr\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.064072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.064043    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.127902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.127865    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:02.374774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.374688    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 00:46:02.374774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.374753    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 00:46:02.588160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.587617    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 00:46:02.613348 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.613308    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:02.613446 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.613405    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:02.613497 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:02.613451    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:03.428158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.421770    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" event={"ID":"4beff93a-f8ea-42c0-864f-151b9c8d4523","Type":"ContainerStarted","Data":"7263fabc4058ab9ccc4ae7fb65b195e9673c0316aa1922928c10c0d2101123af"}
Nov 01 00:46:03.428158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.421851    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" event={"ID":"4beff93a-f8ea-42c0-864f-151b9c8d4523","Type":"ContainerStarted","Data":"cbe2fd17cdf170bcae035a8a6cd8652a29d77ac92d1015aa98d423fbd67dc4d1"}
Nov 01 00:46:03.481941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.481907    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:46:03.482270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.482246    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:46:03.483752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.483725    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 00:46:03.567303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.567256    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podStartSLOduration=2.567231595 podStartE2EDuration="2.567231595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:03.56666789 +0000 UTC m=+1353.244254833" watchObservedRunningTime="2026-11-01 00:46:03.567231595 +0000 UTC m=+1353.244818540"
Nov 01 00:46:03.975398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.975370    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 00:46:03.979257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:03.979226    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.040159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.037217    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-22d7b2-openshift-default-dockercfg-lhnwj\""
Nov 01 00:46:04.048959 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.048926    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.048973    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4c47\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049022    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049047    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049094    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049212    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049266    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049347    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.049381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.049377    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.090591 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.090552    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 00:46:04.140192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.139484    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:04.142153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.142109    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:04.142300 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.142185    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:04.149936 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.149915    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.149958    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.149980    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4c47\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150010    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150190 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150026    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150190 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150061    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150190 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150168    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150389 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150202    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150389 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150260    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150392    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150581 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150505    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.150694 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.150673    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.151099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.151036    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.151099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.151083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.152843 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.152814    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.152983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.152964    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.214590 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.214559    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4c47\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.268017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.267944    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.294688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.294658    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:04.763901 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.763871    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 00:46:04.765808 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:46:04.765782    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7693e327_337d_4a3f_ad05_1cd9a02796cb.slice/crio-d215878ee1109102ae3e36685ca0b3be9130ab1b79d3fdf508b29e8a7afe9513 WatchSource:0}: Error finding container d215878ee1109102ae3e36685ca0b3be9130ab1b79d3fdf508b29e8a7afe9513: Status 404 returned error can't find the container with id d215878ee1109102ae3e36685ca0b3be9130ab1b79d3fdf508b29e8a7afe9513
Nov 01 00:46:04.771712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.768301    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:04.771712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.768365    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:04.771712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:04.768394    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:46:05.138213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:05.137535    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:05.138213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:05.137608    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:05.486007 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:05.485883    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" event={"ID":"7693e327-337d-4a3f-ad05-1cd9a02796cb","Type":"ContainerStarted","Data":"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"}
Nov 01 00:46:05.486007 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:05.485940    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" event={"ID":"7693e327-337d-4a3f-ad05-1cd9a02796cb","Type":"ContainerStarted","Data":"d215878ee1109102ae3e36685ca0b3be9130ab1b79d3fdf508b29e8a7afe9513"}
Nov 01 00:46:05.621091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:05.621032    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podStartSLOduration=2.621010336 podStartE2EDuration="2.621010336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:05.615411269 +0000 UTC m=+1355.292998218" watchObservedRunningTime="2026-11-01 00:46:05.621010336 +0000 UTC m=+1355.298597282"
Nov 01 00:46:06.135160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:06.131530    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:06.135160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:06.131596    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:06.295209 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:06.295173    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:06.296362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:06.296331    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:06.296481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:06.296387    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:07.131161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:07.128550    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:07.131161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:07.128612    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:07.295667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:07.295636    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:07.296049 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:07.295734    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:08.132161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:08.128536    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:08.132161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:08.128599    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:08.295724 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:08.295153    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:08.295724 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:08.295202    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:09.133160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:09.132232    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 00:46:09.133160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:09.132295    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 00:46:09.295909 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:09.295877    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:09.296330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:09.295929    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:10.140303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:10.139455    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:10.140535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:10.140401    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:10.141249 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:10.141228    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 00:46:10.295221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:10.295187    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:10.295407 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:10.295243    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:11.295349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:11.295316    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 00:46:11.295741 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:11.295371    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 00:46:12.299753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:12.299720    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:12.300149 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:12.300097    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:12.300866 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:12.300848    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 00:46:13.383198 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:13.383167    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:46:14.989180 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:46:14.989110    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:46:15.862036 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:15.862003    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 00:46:15.866868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:15.866851    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:15.950782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:15.950752    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 00:46:15.971697 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:15.971664    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"distinct-w3-5a9c19-kserve-self-signed-certs\""
Nov 01 00:46:16.040232 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.040191    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.051998 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.051966    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.052365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.052346    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.052560 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.052539    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.052696 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.052598    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.052696 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.052621    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czqv\" (UniqueName: \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153443 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153366    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153443 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153400    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czqv\" (UniqueName: \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153490    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153541    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153570    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153605    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153880    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.153965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.154026 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.153979    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.155861 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.155839    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.156125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.156109    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.214815 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.214789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czqv\" (UniqueName: \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\") pod \"distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.479352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.479321    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:16.857479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:16.857453    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 00:46:16.859029 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:46:16.859000    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d132881_21f5_425f_b2c3_81af132525d2.slice/crio-d3826d807f65330a5cbe21a9cd49db5689d385be455a1b41b7208f29965d4354 WatchSource:0}: Error finding container d3826d807f65330a5cbe21a9cd49db5689d385be455a1b41b7208f29965d4354: Status 404 returned error can't find the container with id d3826d807f65330a5cbe21a9cd49db5689d385be455a1b41b7208f29965d4354
Nov 01 00:46:17.580580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:17.580543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" event={"ID":"2d132881-21f5-425f-b2c3-81af132525d2","Type":"ContainerStarted","Data":"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"}
Nov 01 00:46:17.581027 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:17.580596    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" event={"ID":"2d132881-21f5-425f-b2c3-81af132525d2","Type":"ContainerStarted","Data":"d3826d807f65330a5cbe21a9cd49db5689d385be455a1b41b7208f29965d4354"}
Nov 01 00:46:17.699008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:17.698960    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podStartSLOduration=2.698945844 podStartE2EDuration="2.698945844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:17.698114648 +0000 UTC m=+1367.375701594" watchObservedRunningTime="2026-11-01 00:46:17.698945844 +0000 UTC m=+1367.376532789"
Nov 01 00:46:18.479631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:18.479598    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:26.480319 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:26.480281    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:26.492975 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:26.492950    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:26.640014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:26.639989    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:46:33.153750 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.153720    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 00:46:33.157869 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.157852    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.260402 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.260370    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"distinct-w2-4143b7-kserve-self-signed-certs\""
Nov 01 00:46:33.263072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263049    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 00:46:33.263583 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263568    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.263633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263591    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.263633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263613    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.263719 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263636    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvkz\" (UniqueName: \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.263778 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.263917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.263905    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.364857 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.364820    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.364865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.364899    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.364923    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.364954    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvkz\" (UniqueName: \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.365209    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365382 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.365359    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.365388    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.365796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.365771    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.367886 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.367849    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.368101 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.368081    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.426705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.426634    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvkz\" (UniqueName: \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\") pod \"distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.469952 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.469923    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:33.847187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:33.847167    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 00:46:33.848284 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:46:33.848261    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod923b9d82_3b07_4d40_99ef_2a0928488985.slice/crio-0021bf6aa53d351a8d6f8f9c217096259acc216e00c003bf7f6e5d955621c6fa WatchSource:0}: Error finding container 0021bf6aa53d351a8d6f8f9c217096259acc216e00c003bf7f6e5d955621c6fa: Status 404 returned error can't find the container with id 0021bf6aa53d351a8d6f8f9c217096259acc216e00c003bf7f6e5d955621c6fa
Nov 01 00:46:34.678286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:34.678241    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" event={"ID":"923b9d82-3b07-4d40-99ef-2a0928488985","Type":"ContainerStarted","Data":"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"}
Nov 01 00:46:34.678821 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:34.678296    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" event={"ID":"923b9d82-3b07-4d40-99ef-2a0928488985","Type":"ContainerStarted","Data":"0021bf6aa53d351a8d6f8f9c217096259acc216e00c003bf7f6e5d955621c6fa"}
Nov 01 00:46:34.795618 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:34.795557    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podStartSLOduration=1.795541974 podStartE2EDuration="1.795541974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:34.795011848 +0000 UTC m=+1384.472598793" watchObservedRunningTime="2026-11-01 00:46:34.795541974 +0000 UTC m=+1384.473128920"
Nov 01 00:46:35.470528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:35.470499    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:43.470596 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:43.470513    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:43.483076 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:43.483046    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:43.739152 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:43.739054    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:46:44.989591 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:46:44.989558    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:46:50.016967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:50.016926    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:46:58.707404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.707369    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 00:46:58.711445 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.711424    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.770764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.770734    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 00:46:58.789470 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789437    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.789647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789564    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnl8c\" (UniqueName: \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.789647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789597    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.789647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789634    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.789812 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789768    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.789864 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.789824    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.813994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.813970    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"distinct-w6-22d7b2-kserve-self-signed-certs\""
Nov 01 00:46:58.814219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.814020    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"openshift-service-ca.crt\""
Nov 01 00:46:58.814219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.814064    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"default-dockercfg-xn6ld\""
Nov 01 00:46:58.814219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.814106    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"kube-root-ca.crt\""
Nov 01 00:46:58.893888 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.893835    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.894094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.893954    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.894094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.894009    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.894094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.894082    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnl8c\" (UniqueName: \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.894346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.894119    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.894346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.894204    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.896933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.894675    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.898548 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.898517    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.898734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.898713    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.900595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.900570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.901670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.901648    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:58.959973 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:58.959895    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnl8c\" (UniqueName: \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\") pod \"distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:59.024643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:59.024612    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:46:59.425158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:59.425114    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 00:46:59.426370 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:46:59.426340    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac707d00_4683_4bc6_a202_4d6e69adc8cf.slice/crio-0968711c4e1ce83447c1d58badcc15cb0a47cb99b27544b6b5f45e9bb1685ea9 WatchSource:0}: Error finding container 0968711c4e1ce83447c1d58badcc15cb0a47cb99b27544b6b5f45e9bb1685ea9: Status 404 returned error can't find the container with id 0968711c4e1ce83447c1d58badcc15cb0a47cb99b27544b6b5f45e9bb1685ea9
Nov 01 00:46:59.826107 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:59.826076    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" event={"ID":"ac707d00-4683-4bc6-a202-4d6e69adc8cf","Type":"ContainerStarted","Data":"b4722915f64b2ebdfddef77257c77a458c8bd67313266bb7bcb55b3baf585ccd"}
Nov 01 00:46:59.826107 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:59.826112    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" event={"ID":"ac707d00-4683-4bc6-a202-4d6e69adc8cf","Type":"ContainerStarted","Data":"0968711c4e1ce83447c1d58badcc15cb0a47cb99b27544b6b5f45e9bb1685ea9"}
Nov 01 00:46:59.943831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:46:59.943776    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podStartSLOduration=1.943760494 podStartE2EDuration="1.943760494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:59.942728973 +0000 UTC m=+1409.620315918" watchObservedRunningTime="2026-11-01 00:46:59.943760494 +0000 UTC m=+1409.621347438"
Nov 01 00:47:00.024964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:00.024936    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:47:09.025500 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:09.025468    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:47:09.038355 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:09.038327    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:47:09.908494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:09.908468    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:47:14.991282 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:47:14.991235    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:47:17.673535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:17.673500    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:47:44.995328 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:47:44.995291    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:47:49.783360 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.783330    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 00:47:49.787098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.787078    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888076 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888045    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888101    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888237    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888303    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888418 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888400    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888426    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888446    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888554    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7b8w\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.888612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.888579    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.894535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.894508    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 00:47:49.990280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990247    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990304    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990322    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990339    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990498    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7b8w\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990558    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990619    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990816 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990700    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990816 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990713    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990816 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990744    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.990816 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990786    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.991018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990968    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.991018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.990999    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.991762 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.991733    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.992912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.992894    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:49.993168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:49.993148    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:50.052103 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.052039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:50.052266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.052127    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7b8w\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:50.100467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.100434    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:50.533423 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.533400    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 00:47:50.535856 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:47:50.535834    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6cc26a_190f_4eb2_8146_0069933a0456.slice/crio-e0873a010f84ad60ee977bd772ecf6186893b6db431ea716d3554ab7d0ab5a7c WatchSource:0}: Error finding container e0873a010f84ad60ee977bd772ecf6186893b6db431ea716d3554ab7d0ab5a7c: Status 404 returned error can't find the container with id e0873a010f84ad60ee977bd772ecf6186893b6db431ea716d3554ab7d0ab5a7c
Nov 01 00:47:50.538187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.538159    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:50.538263 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.538218    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:50.538263 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:50.538251    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:51.144790 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.144753    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" event={"ID":"7d6cc26a-190f-4eb2-8146-0069933a0456","Type":"ContainerStarted","Data":"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"}
Nov 01 00:47:51.145234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.144808    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" event={"ID":"7d6cc26a-190f-4eb2-8146-0069933a0456","Type":"ContainerStarted","Data":"e0873a010f84ad60ee977bd772ecf6186893b6db431ea716d3554ab7d0ab5a7c"}
Nov 01 00:47:51.267740 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.267685    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podStartSLOduration=2.267670968 podStartE2EDuration="2.267670968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:51.264845693 +0000 UTC m=+1460.942432641" watchObservedRunningTime="2026-11-01 00:47:51.267670968 +0000 UTC m=+1460.945257992"
Nov 01 00:47:51.341358 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.341321    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:47:51.345320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.345294    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.389397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.389368    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:47:51.398388 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.398303    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-43a0cf-openshift-default-dockercfg-9jwvj\""
Nov 01 00:47:51.405659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.405624    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.405831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.405749    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.405831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.405811    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.405957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.405847    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.406051 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.406031    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.406126 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.406108    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9f9z\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.406201 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.406184    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.406241 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.406206    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.406398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.406376    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507195 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507159    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9f9z\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507237    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507265    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507328    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507389    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507427    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507494    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507535    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507593    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507812 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507696    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507812 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507752    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.507985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.507970    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.508199 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.508176    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.509913 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.509892    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.510052 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.510026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.568613 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.568584    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.568768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.568661    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9f9z\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:51.659988 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:51.659914    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:52.100999 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.100972    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:52.102370 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.102346    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:52.102467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.102392    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:52.110341 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.110322    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:47:52.112080 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:47:52.112053    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14011b58_4d69_4210_985a_485921ffe69e.slice/crio-0f886b115a53747737e784b8af3d9f0588543b46856e9ee42bdb52e517b2a131 WatchSource:0}: Error finding container 0f886b115a53747737e784b8af3d9f0588543b46856e9ee42bdb52e517b2a131: Status 404 returned error can't find the container with id 0f886b115a53747737e784b8af3d9f0588543b46856e9ee42bdb52e517b2a131
Nov 01 00:47:52.114734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.114699    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:52.114858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.114782    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:52.114858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.114828    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:47:52.156373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:52.156334    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" event={"ID":"14011b58-4d69-4210-985a-485921ffe69e","Type":"ContainerStarted","Data":"0f886b115a53747737e784b8af3d9f0588543b46856e9ee42bdb52e517b2a131"}
Nov 01 00:47:53.101582 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.101544    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:53.101795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.101614    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:53.172902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.172864    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" event={"ID":"14011b58-4d69-4210-985a-485921ffe69e","Type":"ContainerStarted","Data":"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"}
Nov 01 00:47:53.402929 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.402826    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podStartSLOduration=2.402810864 podStartE2EDuration="2.402810864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:53.351763623 +0000 UTC m=+1463.029350562" watchObservedRunningTime="2026-11-01 00:47:53.402810864 +0000 UTC m=+1463.080397850"
Nov 01 00:47:53.661214 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.660643    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:53.664242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.663234    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:53.664242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:53.663301    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:54.101329 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:54.101301    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:54.101572 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:54.101551    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:54.665161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:54.663596    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:54.665161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:54.663673    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:55.101617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:55.101583    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:55.101803 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:55.101661    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:55.665531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:55.665495    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:55.666331 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:55.665970    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:56.101085 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:56.101055    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:56.101316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:56.101295    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:56.661014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:56.660975    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:56.661230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:56.661057    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:57.101106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:57.101065    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 00:47:57.101486 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:57.101149    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 00:47:57.660609 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:57.660574    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:57.660813 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:57.660628    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:58.105556 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:58.105528    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:58.106044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:58.105902    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:58.106552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:58.106532    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 00:47:58.660386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:58.660355    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 00:47:58.660573 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:58.660409    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 00:47:59.417491 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:59.417444    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:47:59.681654 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:59.681572    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:59.683289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:59.683268    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:47:59.686422 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:47:59.686399    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:48:14.990437 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:48:14.990354    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:48:41.881287 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:41.881252    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:48:44.989910 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:48:44.989874    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:48:51.794547 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.794518    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:48:51.798739 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.798718    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.863558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.863524    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:48:51.962149 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962097    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962257    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962393    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962533 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962511    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962591 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962537    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962646 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962622    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962701 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962664    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:51.962748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:51.962725    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066407    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066597    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066656    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066719    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066752    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066834    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066859    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.066893    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.067768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.067414    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.068457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.068353    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.068625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.068600    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.069352 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.069329    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.069736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.069715    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.079240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.079185    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.081763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.081736    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.133575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.133540    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.134820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.134797    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.411903 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.411807    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:52.844179 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.844150    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:48:52.844994 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:48:52.844966    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-188cfcc3a651226aede78bff9f9d94e2f13d3c600d533a490889357aafab9930 WatchSource:0}: Error finding container 188cfcc3a651226aede78bff9f9d94e2f13d3c600d533a490889357aafab9930: Status 404 returned error can't find the container with id 188cfcc3a651226aede78bff9f9d94e2f13d3c600d533a490889357aafab9930
Nov 01 00:48:52.847433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.847404    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:48:52.847529 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.847483    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:48:52.847593 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:52.847527    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:48:53.553419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:53.553387    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" event={"ID":"703227a1-8bce-4dc7-a355-bb79f00312d8","Type":"ContainerStarted","Data":"10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74"}
Nov 01 00:48:53.553621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:53.553434    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" event={"ID":"703227a1-8bce-4dc7-a355-bb79f00312d8","Type":"ContainerStarted","Data":"188cfcc3a651226aede78bff9f9d94e2f13d3c600d533a490889357aafab9930"}
Nov 01 00:48:53.764002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:53.763942    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podStartSLOduration=2.7639276759999998 podStartE2EDuration="2.763927676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:53.711558016 +0000 UTC m=+1523.389144963" watchObservedRunningTime="2026-11-01 00:48:53.763927676 +0000 UTC m=+1523.441514680"
Nov 01 00:48:54.411937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:54.411899    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:48:54.413744 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:54.413716    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:54.413867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:54.413773    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:48:55.412075 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:55.412046    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:55.412478 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:55.412108    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:48:56.412665 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:56.412630    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:56.413050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:56.412680    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:48:57.387274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.387234    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 00:48:57.391302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.391285    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.413019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.412992    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:57.413364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.413046    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:48:57.496452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.496426    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"distinct-2-w2-4143b7-kserve-self-signed-certs\""
Nov 01 00:48:57.499479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.499458    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 00:48:57.564211 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564168    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.564381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564241    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.564381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.564488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564377    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp64t\" (UniqueName: \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.564488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564403    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.564488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.564426    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666115    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666204    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666322    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666355    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp64t\" (UniqueName: \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666379    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666419    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.666955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.666934    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.667795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.667771    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.668271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.668248    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.669694 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.669668    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.669772 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.669695    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:57.743754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:57.743724    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp64t\" (UniqueName: \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\") pod \"distinct-2-w2-4143b7-kserve-778c487cd9-qsknf\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") " pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:58.003948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.003913    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:58.410394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.410364    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 00:48:58.410566 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:48:58.410546    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6408e85_6f99_44ec_865c_c721fa95f958.slice/crio-95eebcf0b285d1ee37d3cda800e89ce4a4afbbc07f07166cad861cbc31e81e50 WatchSource:0}: Error finding container 95eebcf0b285d1ee37d3cda800e89ce4a4afbbc07f07166cad861cbc31e81e50: Status 404 returned error can't find the container with id 95eebcf0b285d1ee37d3cda800e89ce4a4afbbc07f07166cad861cbc31e81e50
Nov 01 00:48:58.412252 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.412229    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:58.412317 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.412295    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:48:58.638164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.637839    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" event={"ID":"b6408e85-6f99-44ec-865c-c721fa95f958","Type":"ContainerStarted","Data":"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"}
Nov 01 00:48:58.638164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:58.637898    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" event={"ID":"b6408e85-6f99-44ec-865c-c721fa95f958","Type":"ContainerStarted","Data":"95eebcf0b285d1ee37d3cda800e89ce4a4afbbc07f07166cad861cbc31e81e50"}
Nov 01 00:48:59.004550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:59.004514    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:48:59.412314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:59.412220    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 00:48:59.412496 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:48:59.412302    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 00:49:00.417328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:00.417299    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:49:00.417785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:00.417747    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:49:00.418369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:00.418351    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:49:00.642578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:00.642523    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podStartSLOduration=3.642508655 podStartE2EDuration="3.642508655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:58.765421398 +0000 UTC m=+1528.443008344" watchObservedRunningTime="2026-11-01 00:49:00.642508655 +0000 UTC m=+1530.320095651"
Nov 01 00:49:08.004361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:08.004315    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:49:08.018035 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:08.018001    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:49:08.751630 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:08.751601    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:49:14.989970 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:49:14.989934    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:49:21.073436 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.073398    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:49:21.077364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.077343    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.185717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.185688    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:49:21.245517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245483    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njs65\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245686 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245522    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245686 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245612    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245758 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245719    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245856 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245842    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245878    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245930 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245898    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.245983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.245938    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.246058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.246044    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347069    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347163    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347202    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347228    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347254    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347358    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njs65\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347385    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347428    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347508    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347734    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.347954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347933    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.348029 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347979    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.348115 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.347999    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.349799 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.349780    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.349934 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.349917    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.409351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.409310    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.409520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.409495    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njs65\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:21.692340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:21.692252    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:22.132173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.132120    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:49:22.133741 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:49:22.133716    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2391d5_b464_416c_860a_6eee28e98be7.slice/crio-b66c532c8678a60b0104697316862a719892073989a2900754648eb7cd4f470a WatchSource:0}: Error finding container b66c532c8678a60b0104697316862a719892073989a2900754648eb7cd4f470a: Status 404 returned error can't find the container with id b66c532c8678a60b0104697316862a719892073989a2900754648eb7cd4f470a
Nov 01 00:49:22.136762 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.136723    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:22.136850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.136810    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:22.136892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.136857    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:22.846481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.846446    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" event={"ID":"1b2391d5-b464-416c-860a-6eee28e98be7","Type":"ContainerStarted","Data":"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"}
Nov 01 00:49:22.846687 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:22.846508    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" event={"ID":"1b2391d5-b464-416c-860a-6eee28e98be7","Type":"ContainerStarted","Data":"b66c532c8678a60b0104697316862a719892073989a2900754648eb7cd4f470a"}
Nov 01 00:49:23.022690 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:23.022626    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podStartSLOduration=2.022603003 podStartE2EDuration="2.022603003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:23.016772259 +0000 UTC m=+1552.694359201" watchObservedRunningTime="2026-11-01 00:49:23.022603003 +0000 UTC m=+1552.700189949"
Nov 01 00:49:23.692693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:23.692660    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:23.694417 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:23.694390    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:23.694534 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:23.694449    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:24.693232 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:24.693202    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:24.693660 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:24.693255    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:25.693413 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:25.693375    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:25.693811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:25.693443    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:26.693260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:26.693226    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:26.693470 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:26.693286    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:27.693036 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:27.692996    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:27.693280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:27.693063    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:28.692913 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:28.692872    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 00:49:28.693432 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:28.692938    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 00:49:29.697599 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:29.697562    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:29.698072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:29.697930    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:29.698945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:29.698918    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:49:31.818639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:31.818609    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:49:34.373442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.373399    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.404165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.403981    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.473928 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.473870    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.766159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.740450    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.766159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.740811    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:49:34.802160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.801572    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.866404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.866338    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:34.906647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:34.906598    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.090333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.090227    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.181278 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.181231    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.375333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.375224    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:49:35.376558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.376535    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:49:35.423693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.418233    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.423693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.418642    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:49:35.532163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.530807    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.628212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.628106    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:49:35.747416 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.747373    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:49:35.764165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.749050    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:49:35.819763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.819722    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.820751 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.820729    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:49:35.867233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.866795    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:49:35.868292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.867527    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:49:35.939308 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.930195    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:49:35.939308 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.931654    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:49:35.939308 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:35.932047    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:49:36.008879 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:36.008846    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:49:36.030043 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:36.030014    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:49:36.030537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:36.030509    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:49:36.031184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:36.031165    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:49:44.990759 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:49:44.990676    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:49:51.972408 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:51.972373    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 00:49:51.975867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:51.975842    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.022659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.022628    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 00:49:52.026771 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.026753    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073127    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073185    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073255    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073291    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073321    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073425    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073455    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073530    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073548    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073663    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073682    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073697    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgpp2\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073714    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.073826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073740    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zth5s\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073787    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.073826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.073817    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.074095 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.074067    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.074321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.074190    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.077922 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.077899    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 00:49:52.130632 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.130593    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 00:49:52.175748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.175694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.175945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.175762    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.175945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.175843    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.175945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.175871    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.175993    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176017    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176041    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgpp2\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176072    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176112    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zth5s\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176187    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176203    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176231    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176275    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176316    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176338    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176353    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176349    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176373    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.176452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176432    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.177019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176775    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.177019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.176861    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.177175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.177116    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.177445 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.177417    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.177537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.177428    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.177753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.177728    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.178073 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.178047    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.178304 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.177048    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.178464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.178442    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.179240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.179090    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.180064 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.180033    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.180227 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.180210    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.180587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.180572    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.238649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.238578    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgpp2\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.239439 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.239412    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zth5s\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.240147 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.240080    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.241747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.241722    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.289549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.289507    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:52.305526 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.305493    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 00:49:52.309988 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.309969    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.342964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.342697    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:52.374178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.373783    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 00:49:52.379917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.379892    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.379981 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.379955    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380017 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380000    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380067 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380054    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380103 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380089    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380202    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380318 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380308    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.380351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.380332    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6jz\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481513    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481713 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481569    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481800 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481850    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481930 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481902    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.481930 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.481922    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6jz\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482038 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482012    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482038 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482016    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482047    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482046    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482074    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482543 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482523    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482641 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482576    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.482737 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.482717    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.484727 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.484704    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.485242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.485224    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.543436 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.543365    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.543563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.543442    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6jz\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.627450 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.627421    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:52.757672 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.757650    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 00:49:52.760151 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:49:52.760104    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdab793_77bc_4daa_aefd_80b03bd3277f.slice/crio-6aea9a911412c7f49a6562dcd1974527e2a5510691028e2dbdf1b6af816bd239 WatchSource:0}: Error finding container 6aea9a911412c7f49a6562dcd1974527e2a5510691028e2dbdf1b6af816bd239: Status 404 returned error can't find the container with id 6aea9a911412c7f49a6562dcd1974527e2a5510691028e2dbdf1b6af816bd239
Nov 01 00:49:52.762731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.762697    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:52.762832 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.762779    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:52.762832 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.762825    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:52.794535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.794507    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 00:49:52.796020 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:49:52.795995    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c801fe_588b_4c9d_8f42_7c32f2556346.slice/crio-7648a524b04b6591b28e79a4cc8ac7880d390b939766b0fcc012f89d753654d7 WatchSource:0}: Error finding container 7648a524b04b6591b28e79a4cc8ac7880d390b939766b0fcc012f89d753654d7: Status 404 returned error can't find the container with id 7648a524b04b6591b28e79a4cc8ac7880d390b939766b0fcc012f89d753654d7
Nov 01 00:49:52.798008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.797979    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:52.798082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.798042    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:52.798082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:52.798072    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:53.015265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.014879    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 00:49:53.018941 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:49:53.018900    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfddc601_9e1f_452e_bb4f_9e3441f35902.slice/crio-3210b58381bb30df46e88ddcccb2110ad6f7c16bb4ec4172f8b41a84b7f804f7 WatchSource:0}: Error finding container 3210b58381bb30df46e88ddcccb2110ad6f7c16bb4ec4172f8b41a84b7f804f7: Status 404 returned error can't find the container with id 3210b58381bb30df46e88ddcccb2110ad6f7c16bb4ec4172f8b41a84b7f804f7
Nov 01 00:49:53.022831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.022787    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:53.022943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.022888    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:53.022943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.022935    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:49:53.179089 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.179045    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" event={"ID":"0fdab793-77bc-4daa-aefd-80b03bd3277f","Type":"ContainerStarted","Data":"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"}
Nov 01 00:49:53.179303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.179102    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" event={"ID":"0fdab793-77bc-4daa-aefd-80b03bd3277f","Type":"ContainerStarted","Data":"6aea9a911412c7f49a6562dcd1974527e2a5510691028e2dbdf1b6af816bd239"}
Nov 01 00:49:53.181884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.181859    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" event={"ID":"cfddc601-9e1f-452e-bb4f-9e3441f35902","Type":"ContainerStarted","Data":"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"}
Nov 01 00:49:53.182014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.181899    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" event={"ID":"cfddc601-9e1f-452e-bb4f-9e3441f35902","Type":"ContainerStarted","Data":"3210b58381bb30df46e88ddcccb2110ad6f7c16bb4ec4172f8b41a84b7f804f7"}
Nov 01 00:49:53.184672 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.184629    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" event={"ID":"96c801fe-588b-4c9d-8f42-7c32f2556346","Type":"ContainerStarted","Data":"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"}
Nov 01 00:49:53.184778 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.184694    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" event={"ID":"96c801fe-588b-4c9d-8f42-7c32f2556346","Type":"ContainerStarted","Data":"7648a524b04b6591b28e79a4cc8ac7880d390b939766b0fcc012f89d753654d7"}
Nov 01 00:49:53.290994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.290460    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:49:53.295908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.295479    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:53.295908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.295544    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:53.307207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.304940    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podStartSLOduration=2.304921185 podStartE2EDuration="2.304921185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.300499233 +0000 UTC m=+1582.978086205" watchObservedRunningTime="2026-11-01 00:49:53.304921185 +0000 UTC m=+1582.982508151"
Nov 01 00:49:53.346156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.345364    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:49:53.346156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.345527    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:53.346156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.345572    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:53.564125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.563379    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podStartSLOduration=1.563357581 podStartE2EDuration="1.563357581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.560687192 +0000 UTC m=+1583.238274137" watchObservedRunningTime="2026-11-01 00:49:53.563357581 +0000 UTC m=+1583.240944539"
Nov 01 00:49:53.575161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.570938    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podStartSLOduration=2.570918448 podStartE2EDuration="2.570918448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.437487567 +0000 UTC m=+1583.115074524" watchObservedRunningTime="2026-11-01 00:49:53.570918448 +0000 UTC m=+1583.248505393"
Nov 01 00:49:53.631359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:53.631328    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:49:54.291091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.290550    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:54.291091 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.290611    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:54.354167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.343586    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:54.354167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.343647    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:54.635213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.634205    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:54.635213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:54.634265    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:49:55.291157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.290611    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:55.291157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.290681    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:55.344156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.343614    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:55.344156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.343674    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:55.642219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.628546    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:55.642219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:55.628610    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:49:56.294158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.293651    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:56.294158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.293711    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:56.344158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.343588    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:56.344158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.343653    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:56.634219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.631443    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:56.634219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:56.631502    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:49:57.292245 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.291756    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:57.292245 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.291822    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:57.348160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.345553    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:57.348160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.345618    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:57.629479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.629384    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:57.629746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:57.629718    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:49:58.293175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.293043    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:58.293175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.293117    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:58.351170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.350370    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:58.351170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.350426    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:58.637219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.635271    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:58.637219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:58.635334    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:49:59.290609 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.290569    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 00:49:59.290760 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.290631    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 00:49:59.345156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.344528    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 00:49:59.345156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.344588    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 00:49:59.628268 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.628165    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 00:49:59.628268 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:49:59.628228    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 00:50:00.575949 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.575919    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:50:00.578552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.577246    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:50:00.578940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.578904    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 00:50:00.661365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.655841    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:50:00.661365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.657107    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:50:00.661365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.660575    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 00:50:00.808485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.808451    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:50:00.809739 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.809713    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:50:00.810501 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:00.810478    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 00:50:02.633772 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.633737    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:50:02.639445 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.639421    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681317    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681385    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681484    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681599 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681539    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblpt\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681660    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681699    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681762 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681730    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681801 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681784    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.681896 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.681873    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.682089 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.682074    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 00:50:02.688851 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.688824    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.744203 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.744171    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 00:50:02.747927 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.747899    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:50:02.782827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.782785    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.782827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.782836    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.782888    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblpt\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.782903    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783043    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783082 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783069    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783102    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783117    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783156    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783173    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783197    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783212    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783270    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsqt\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783275    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783295    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783362    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.783897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783400    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.783897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783476    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.784007 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.783971    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.784249 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.784190    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.784390 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.784370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.784734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.784709    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.786145 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.786110    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.786255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.786110    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.844468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.844429    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.844608 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.844580    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblpt\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:02.884652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884554    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884652    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884684    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884715    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884750    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsqt\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884778    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.884947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.884852    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885024    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885086    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885417 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885352    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885417 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885373    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.885973 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.885949    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.887167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.887123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.887511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.887490    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.945464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.945428    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.945751 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.945730    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsqt\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:02.954624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:02.954595    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:03.020469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.020425    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:03.386559 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.386530    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:50:03.387703 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:03.387676    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63079b0_5263_47db_9bb0_41eb3d415ca0.slice/crio-de86be7e8b95a897e7193e13df56c7acbd7fcfc9bba0c81b4a9d4e5875e10429 WatchSource:0}: Error finding container de86be7e8b95a897e7193e13df56c7acbd7fcfc9bba0c81b4a9d4e5875e10429: Status 404 returned error can't find the container with id de86be7e8b95a897e7193e13df56c7acbd7fcfc9bba0c81b4a9d4e5875e10429
Nov 01 00:50:03.390220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.390189    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:03.390295 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.390263    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:03.390327 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.390295    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:03.440706 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.440678    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 00:50:03.441546 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:03.441521    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1e12db_6952_45d4_9cfc_609e9e8dd4f3.slice/crio-3ba93dc0ef864faf2ac069bb9fe61b80909f46a37b9c829e974cc1c5bea9ecee WatchSource:0}: Error finding container 3ba93dc0ef864faf2ac069bb9fe61b80909f46a37b9c829e974cc1c5bea9ecee: Status 404 returned error can't find the container with id 3ba93dc0ef864faf2ac069bb9fe61b80909f46a37b9c829e974cc1c5bea9ecee
Nov 01 00:50:03.444197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.444110    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:03.444302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.444252    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:03.444367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:03.444313    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:04.305848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.305799    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" event={"ID":"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3","Type":"ContainerStarted","Data":"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"}
Nov 01 00:50:04.306457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.305908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" event={"ID":"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3","Type":"ContainerStarted","Data":"3ba93dc0ef864faf2ac069bb9fe61b80909f46a37b9c829e974cc1c5bea9ecee"}
Nov 01 00:50:04.308815 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.308778    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" event={"ID":"c63079b0-5263-47db-9bb0-41eb3d415ca0","Type":"ContainerStarted","Data":"170702f1870773dc46f820970c78cd0791c254d6cbc2b84eb906946ab06be7b3"}
Nov 01 00:50:04.308940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.308870    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" event={"ID":"c63079b0-5263-47db-9bb0-41eb3d415ca0","Type":"ContainerStarted","Data":"de86be7e8b95a897e7193e13df56c7acbd7fcfc9bba0c81b4a9d4e5875e10429"}
Nov 01 00:50:04.593550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.593441    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podStartSLOduration=2.593426981 podStartE2EDuration="2.593426981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.591614218 +0000 UTC m=+1594.269201163" watchObservedRunningTime="2026-11-01 00:50:04.593426981 +0000 UTC m=+1594.271014023"
Nov 01 00:50:04.595128 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.595094    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podStartSLOduration=2.595084033 podStartE2EDuration="2.595084033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.470786715 +0000 UTC m=+1594.148373662" watchObservedRunningTime="2026-11-01 00:50:04.595084033 +0000 UTC m=+1594.272670980"
Nov 01 00:50:04.961208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.955516    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:04.961208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.958410    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:04.961208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:04.958468    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:05.044159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.044101    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:05.044374 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.044285    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:05.044441 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.044367    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:05.872748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.872706    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 00:50:05.877065 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.877044    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.959293 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.959233    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:05.959567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.959536    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974180    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974264    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974371    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974488    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974637    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974662    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974686    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974763    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:05.975181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:05.974829    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6k6\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.061360 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.061322    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:06.073776 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.073733    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.077965    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078068    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078093    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078116    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078198    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6k6\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078351    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.078455    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.079520    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.080015    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.080255    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.080663    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.080920    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.084935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.082517    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.086797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.086767    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.096296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.092711    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 00:50:06.174929 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.174849    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.176274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.176255    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6k6\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.191437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.191416    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:06.678938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.678896    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 00:50:06.679815 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:06.679794    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34be36_7988_407b_a370_82f7cfb0d75e.slice/crio-355e53800ffc2bf1130b5484626ed7beac640e65cbe29c038fa9fef0b0cba7b8 WatchSource:0}: Error finding container 355e53800ffc2bf1130b5484626ed7beac640e65cbe29c038fa9fef0b0cba7b8: Status 404 returned error can't find the container with id 355e53800ffc2bf1130b5484626ed7beac640e65cbe29c038fa9fef0b0cba7b8
Nov 01 00:50:06.683145 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.683091    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:06.683334 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.683309    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:06.683490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.683457    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:06.964215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.963684    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:06.964215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:06.963743    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:07.038163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.037368    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:07.038163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.037426    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:07.383887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.383856    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" event={"ID":"ae34be36-7988-407b-a370-82f7cfb0d75e","Type":"ContainerStarted","Data":"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"}
Nov 01 00:50:07.384041 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.383907    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" event={"ID":"ae34be36-7988-407b-a370-82f7cfb0d75e","Type":"ContainerStarted","Data":"355e53800ffc2bf1130b5484626ed7beac640e65cbe29c038fa9fef0b0cba7b8"}
Nov 01 00:50:07.514888 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.514838    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podStartSLOduration=2.514824375 podStartE2EDuration="2.514824375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:07.510606262 +0000 UTC m=+1597.188193219" watchObservedRunningTime="2026-11-01 00:50:07.514824375 +0000 UTC m=+1597.192411319"
Nov 01 00:50:07.972169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.971581    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:07.972169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:07.971641    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:08.040157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.036570    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:08.040157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.036627    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:08.192501 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.192472    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:08.195897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.195278    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:08.195897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.195335    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:08.957627 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.957592    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:08.958356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:08.958325    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:09.024627 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.024587    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:09.025240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.025207    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:09.194166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.193615    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:09.194166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.193693    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:09.964628 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.964591    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:09.964978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:09.964937    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:10.026596 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.026560    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:10.027212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.027179    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:10.192120 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.192085    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:10.192400 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.192361    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:10.957511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.957013    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 00:50:10.957511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:10.957080    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 00:50:11.024160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.021483    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 00:50:11.024160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.021554    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 00:50:11.192764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.192718    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:11.193588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.193555    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:11.959673 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.959645    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:11.960046 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.960029    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:11.960706 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:11.960688    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:50:12.043820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:12.043792    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:12.044302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:12.044187    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:12.045220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:12.045200    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 00:50:12.192527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:12.192489    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:12.192692 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:12.192545    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:13.052419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:13.052384    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"]
Nov 01 00:50:13.056320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:13.056292    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64579a35-sb2f5"]
Nov 01 00:50:13.192404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:13.192368    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 00:50:13.192566 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:13.192427    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 00:50:14.053265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.053234    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"]
Nov 01 00:50:14.055943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.055918    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c6ce84-qnz4m"]
Nov 01 00:50:14.507285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.507241    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfefe2ae-a027-4f21-b742-81c2caa58e21" path="/var/lib/kubelet/pods/bfefe2ae-a027-4f21-b742-81c2caa58e21/volumes"
Nov 01 00:50:14.507995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.507974    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5c344c-a353-4f6b-87c0-254a527f53f5" path="/var/lib/kubelet/pods/ee5c344c-a353-4f6b-87c0-254a527f53f5/volumes"
Nov 01 00:50:14.625779 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.625749    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:14.626730 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.626703    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:14.627618 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:14.627598    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 00:50:14.997913 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:50:14.997876    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:50:17.738159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:17.726763    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:17.948160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:17.942215    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:17.994436 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:17.994333    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:50:18.021851 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.020282    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.062197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.036278    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.062197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.037169    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.091160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.065646    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.120215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.091952    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.185862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.185831    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.207299 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.204003    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253432    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253479    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253526    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xqq\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253641    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253665    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253703    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253747    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253785    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.255735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.253825    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.330781 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.316980    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394523    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394575    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394625    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394811    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394838    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.394890    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xqq\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.395013    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.395040    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.395084    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.397169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.395675    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.438160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.424377    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"]
Nov 01 00:50:18.442259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.439368    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.442465 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.440436    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.442587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.440551    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.442688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.441222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.519625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.510379    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.519625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.512563    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.561165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.551814    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.671156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.649064    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:50:18.718286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.667843    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xqq\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.718286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.701980    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a00cda00-dshzm"]
Nov 01 00:50:18.735272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.735178    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.764995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.735664    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:18.773212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.735901    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:18.773212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:18.768702    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:50:19.079229 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.020496    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.100165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.094545    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:50:19.216194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.037205    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.216194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.037291    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.248278 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.220117    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:50:19.355056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.354978    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:50:19.356597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.129436    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.400694 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.245840    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.466856 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.032653    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:19.467019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.129470    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 01 00:50:19.467019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.403296    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:50:19.467173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.467022    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:50:19.467173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.467084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:50:19.467173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.467160    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:50:19.467335 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.467223    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:50:19.578088 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.578023    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:50:19.588645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.578638    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:50:19.588925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.588875    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:50:19.994309 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.994283    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:50:19.996035 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:19.996005    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cf6c5e_cae8_4e69_b061_2244b8732a49.slice/crio-6ecf8c60d05edd10b3b822787238d85deeb80fc83a2fea494541fd6c84eaa97a WatchSource:0}: Error finding container 6ecf8c60d05edd10b3b822787238d85deeb80fc83a2fea494541fd6c84eaa97a: Status 404 returned error can't find the container with id 6ecf8c60d05edd10b3b822787238d85deeb80fc83a2fea494541fd6c84eaa97a
Nov 01 00:50:19.998718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.998683    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:19.998820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.998762    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:19.998820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:19.998806    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:20.183985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:20.183952    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" event={"ID":"61cf6c5e-cae8-4e69-b061-2244b8732a49","Type":"ContainerStarted","Data":"0d763b801e01851a07954de5ac994bb3df8904b46ffcc46da3c93aacf2f3ea4b"}
Nov 01 00:50:20.184255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:20.184235    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" event={"ID":"61cf6c5e-cae8-4e69-b061-2244b8732a49","Type":"ContainerStarted","Data":"6ecf8c60d05edd10b3b822787238d85deeb80fc83a2fea494541fd6c84eaa97a"}
Nov 01 00:50:20.519810 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:20.519755    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podStartSLOduration=3.519736957 podStartE2EDuration="3.519736957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:20.514351704 +0000 UTC m=+1610.191938650" watchObservedRunningTime="2026-11-01 00:50:20.519736957 +0000 UTC m=+1610.197323903"
Nov 01 00:50:20.521783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:20.521754    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224c039b-fad5-48f9-842f-c63072db840b" path="/var/lib/kubelet/pods/224c039b-fad5-48f9-842f-c63072db840b/volumes"
Nov 01 00:50:20.735848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:20.735817    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:21.757651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:21.757618    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:21.758283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:21.758239    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:22.740661 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:22.740625    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:22.740951 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:22.740921    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:23.741164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:23.737453    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:23.741164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:23.737523    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:24.740554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:24.740522    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:24.740774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:24.740580    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:25.740461 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:25.740415    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:25.740975 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:25.740490    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:26.736301 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:26.736271    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 00:50:26.736481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:26.736321    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 00:50:27.740977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:27.740951    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:27.741454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:27.741446    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:27.742225 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:27.742205    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:50:36.689826 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.689793    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:50:36.695882 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.695860    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.833794 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.833758    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.833985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.833810    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.833985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.833872    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834120 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834002    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834120 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834050    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8wn\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834269    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834380    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834465 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834448    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.834527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.834492    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.901606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.901571    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:50:36.936255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936225    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936278    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936307    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936369    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936394    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8wn\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936522    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936615    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936671    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.936754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.936719    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.937115 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.937092    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.937618 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.937598    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.940620 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.940111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.940726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.940681    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.941251 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.941226    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.941520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.941500    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.943628 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.943606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:36.993596 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.993567    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 00:50:36.997931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:36.997914    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.005584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.005553    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:37.006753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.006733    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8wn\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:37.010541 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.010521    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:37.050875 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.050845    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5a9c19-openshift-default-dockercfg-vrnmb\""
Nov 01 00:50:37.063613 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.063578    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 00:50:37.140001 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.139962    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddz2\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140232 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140034    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140232 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140126    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140223    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140285    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140398    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140441    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140590 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140503    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.140590 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.140557    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.241718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241685    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.241892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241738    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.241892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241839    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddz2\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.241892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241873    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.241892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241891    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242103 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241912    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242103 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.241955    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242103 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242059    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242106    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242285    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242429    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242515    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242560    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.242734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.242593    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.244394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.244375    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.244518 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.244502    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.305825 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.305789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.305972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.305844    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddz2\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.362806 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.362496    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:37.459592 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:37.459564    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd396d4d_601b_45cb_80b0_cd29ab967520.slice/crio-c581bb16b141f01832adb19e4a9415c6526d9fc2b739ac686f1f0b51b1ac0adc WatchSource:0}: Error finding container c581bb16b141f01832adb19e4a9415c6526d9fc2b739ac686f1f0b51b1ac0adc: Status 404 returned error can't find the container with id c581bb16b141f01832adb19e4a9415c6526d9fc2b739ac686f1f0b51b1ac0adc
Nov 01 00:50:37.461196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.460798    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:50:37.462330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.462300    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.462414 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.462366    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.462414 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.462395    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.669866 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.669834    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 00:50:37.673992 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.673971    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.749259 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.749175    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 00:50:37.767999 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.767960    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768025    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgcq\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768196 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768103    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768241    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768284    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768315    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768501    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768528    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.768664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.768649    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.821189 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.821165    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 00:50:37.822624 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:37.822596    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce48cca2_61fa_4da0_b543_3eaf0d86af79.slice/crio-9a72634bf96e8daf2430fe7d3b04e54e169e50eef68477f49e08c0a4398cc735 WatchSource:0}: Error finding container 9a72634bf96e8daf2430fe7d3b04e54e169e50eef68477f49e08c0a4398cc735: Status 404 returned error can't find the container with id 9a72634bf96e8daf2430fe7d3b04e54e169e50eef68477f49e08c0a4398cc735
Nov 01 00:50:37.825637 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.825605    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.825718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.825678    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.825718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.825707    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:37.869506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869478    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869511    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869599    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869671    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869806    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgcq\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869862    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.869942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869939    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869945    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.869991    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.870007    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.870038    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870416 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.870396    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.870576    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.870797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.870778    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.872428 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.872410    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.872706 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.872684    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.933336 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.933296    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.933336 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.933323    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgcq\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:37.989454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:37.989160    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:38.303224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.303148    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" event={"ID":"ce48cca2-61fa-4da0-b543-3eaf0d86af79","Type":"ContainerStarted","Data":"4c8f7dbf0c86b01052f3958ec6b54502fe7036f8e28bb872c3d46477e4efe261"}
Nov 01 00:50:38.303224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.303203    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" event={"ID":"ce48cca2-61fa-4da0-b543-3eaf0d86af79","Type":"ContainerStarted","Data":"9a72634bf96e8daf2430fe7d3b04e54e169e50eef68477f49e08c0a4398cc735"}
Nov 01 00:50:38.305016 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.304989    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" event={"ID":"cd396d4d-601b-45cb-80b0-cd29ab967520","Type":"ContainerStarted","Data":"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"}
Nov 01 00:50:38.305187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.305034    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" event={"ID":"cd396d4d-601b-45cb-80b0-cd29ab967520","Type":"ContainerStarted","Data":"c581bb16b141f01832adb19e4a9415c6526d9fc2b739ac686f1f0b51b1ac0adc"}
Nov 01 00:50:38.363366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.363335    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:38.365057 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.365033    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:38.365210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.365085    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:38.411090 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.411048    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 00:50:38.412780 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:38.412754    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb4879f_f720_4a9d_b9a1_415fdff20edf.slice/crio-a59f208fee85389532f5b28df6e82d869a6dee47133a78ed3a315be575703e37 WatchSource:0}: Error finding container a59f208fee85389532f5b28df6e82d869a6dee47133a78ed3a315be575703e37: Status 404 returned error can't find the container with id a59f208fee85389532f5b28df6e82d869a6dee47133a78ed3a315be575703e37
Nov 01 00:50:38.422706 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.422657    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:38.482164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.439790    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podStartSLOduration=2.439771223 podStartE2EDuration="2.439771223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:38.439062431 +0000 UTC m=+1628.116649401" watchObservedRunningTime="2026-11-01 00:50:38.439771223 +0000 UTC m=+1628.117358171"
Nov 01 00:50:38.482164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.478608    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:38.482164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.478720    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:50:38.577219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:38.577085    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podStartSLOduration=2.577064461 podStartE2EDuration="2.577064461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:38.576520094 +0000 UTC m=+1628.254107042" watchObservedRunningTime="2026-11-01 00:50:38.577064461 +0000 UTC m=+1628.254651412"
Nov 01 00:50:39.011273 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.011225    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:39.013063 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.013029    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:39.013285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.013091    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:39.367215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.362423    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" event={"ID":"cfb4879f-f720-4a9d-b9a1-415fdff20edf","Type":"ContainerStarted","Data":"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"}
Nov 01 00:50:39.367215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.362488    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" event={"ID":"cfb4879f-f720-4a9d-b9a1-415fdff20edf","Type":"ContainerStarted","Data":"a59f208fee85389532f5b28df6e82d869a6dee47133a78ed3a315be575703e37"}
Nov 01 00:50:39.385162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.375365    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:39.385162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.375426    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:39.996441 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.996389    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:39.997851 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.997816    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:39.997991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:39.997877    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:40.011551 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.011524    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:40.011869 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.011571    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:40.394847 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.394754    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:40.395111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.395086    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:40.989867 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.989829    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:40.990218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:40.990190    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:41.010829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.010804    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:41.011083 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.011019    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:41.364637 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.364544    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:41.365234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.365201    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:41.991056 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.991020    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:41.991388 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:41.991358    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:42.011295 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.011268    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:42.011449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.011316    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:42.369909 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.369698    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:42.374330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.374289    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:42.991545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.991509    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:42.991813 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:42.991784    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:43.011073 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.010991    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:43.011073 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.011050    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:43.369712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.369617    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:43.369996 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.369966    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:43.990083 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.990047    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:43.990469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:43.990427    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:44.011339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.011307    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:44.011473 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.011383    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:44.363639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.363563    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 00:50:44.363828 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.363792    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 00:50:44.997160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.995680    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:44.997160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:44.995738    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:45.025211 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.022700    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 00:50:45.025211 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.022778    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 00:50:45.039458 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:50:45.039329    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:50:45.951285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.951245    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:45.952179 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.952150    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:45.952859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.952837    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 00:50:45.989852 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.989817    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 00:50:45.990008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:45.989872    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 00:50:46.016518 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.016488    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:46.016956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.016929    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:46.017732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.017713    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:50:46.055032 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.054981    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podStartSLOduration=9.054967113 podStartE2EDuration="9.054967113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:39.491604276 +0000 UTC m=+1629.169191227" watchObservedRunningTime="2026-11-01 00:50:46.054967113 +0000 UTC m=+1635.732554058"
Nov 01 00:50:46.057941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.057914    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"]
Nov 01 00:50:46.062364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.062342    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.071317 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.071287    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkt7\" (UniqueName: \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.071454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.071432    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.071519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.071481    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.115512 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.115484    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 00:50:46.115512 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.115503    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 00:50:46.118912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.118892    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"]
Nov 01 00:50:46.173066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.173027    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkt7\" (UniqueName: \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.173306 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.173124    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.173306 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.173179    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.174011 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.173988    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.176701 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.176674    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.234482 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.234452    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkt7\" (UniqueName: \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\") pod \"maas-api-revoke-sub-7cd01240-ww5cx\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.385422 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.377396    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:46.827017 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:50:46.826986    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d0c42a_95f8_49e0_b74e_5658509a1c82.slice/crio-23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578 WatchSource:0}: Error finding container 23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578: Status 404 returned error can't find the container with id 23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578
Nov 01 00:50:46.827224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.827200    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"]
Nov 01 00:50:46.994516 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.994484    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:46.994899 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.994879    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:46.995674 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:46.995654    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 00:50:47.509884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:47.509794    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx" event={"ID":"19d0c42a-95f8-49e0-b74e-5658509a1c82","Type":"ContainerStarted","Data":"9339e2d5567032f1d7ea66f2858a5da73990619f3e6a92ce0df05dbfa6177c48"}
Nov 01 00:50:47.509884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:47.509854    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx" event={"ID":"19d0c42a-95f8-49e0-b74e-5658509a1c82","Type":"ContainerStarted","Data":"23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578"}
Nov 01 00:50:47.628594 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:47.628544    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx" podStartSLOduration=1.628529023 podStartE2EDuration="1.628529023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:47.626172111 +0000 UTC m=+1637.303759057" watchObservedRunningTime="2026-11-01 00:50:47.628529023 +0000 UTC m=+1637.306115968"
Nov 01 00:50:48.517549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:48.517513    4493 generic.go:356] "Generic (PLEG): container finished" podID="19d0c42a-95f8-49e0-b74e-5658509a1c82" containerID="9339e2d5567032f1d7ea66f2858a5da73990619f3e6a92ce0df05dbfa6177c48" exitCode=0
Nov 01 00:50:48.517965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:48.517559    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx" event={"ID":"19d0c42a-95f8-49e0-b74e-5658509a1c82","Type":"ContainerDied","Data":"9339e2d5567032f1d7ea66f2858a5da73990619f3e6a92ce0df05dbfa6177c48"}
Nov 01 00:50:49.706256 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.706232    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:49.737449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.737421    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\") pod \"19d0c42a-95f8-49e0-b74e-5658509a1c82\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") "
Nov 01 00:50:49.737602 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.737516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\") pod \"19d0c42a-95f8-49e0-b74e-5658509a1c82\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") "
Nov 01 00:50:49.737669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.737658    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\" (UniqueName: \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\") pod \"19d0c42a-95f8-49e0-b74e-5658509a1c82\" (UID: \"19d0c42a-95f8-49e0-b74e-5658509a1c82\") "
Nov 01 00:50:49.737811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.737776    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca" pod "19d0c42a-95f8-49e0-b74e-5658509a1c82" (UID: "19d0c42a-95f8-49e0-b74e-5658509a1c82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:50:49.738205 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.738188    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:50:49.740566 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.740540    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca" pod "19d0c42a-95f8-49e0-b74e-5658509a1c82" (UID: "19d0c42a-95f8-49e0-b74e-5658509a1c82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:50:49.740662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.740637    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7" pod "19d0c42a-95f8-49e0-b74e-5658509a1c82" (UID: "19d0c42a-95f8-49e0-b74e-5658509a1c82"). InnerVolumeSpecName "kube-api-access-sdkt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:50:49.839736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.839654    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkt7\" (UniqueName: \"kubernetes.io/projected/19d0c42a-95f8-49e0-b74e-5658509a1c82-kube-api-access-sdkt7\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:50:49.839736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:49.839685    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d0c42a-95f8-49e0-b74e-5658509a1c82-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:50:50.533168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:50.533124    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"
Nov 01 00:50:50.533321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:50.533166    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx" event={"ID":"19d0c42a-95f8-49e0-b74e-5658509a1c82","Type":"ContainerDied","Data":"23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578"}
Nov 01 00:50:50.533321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:50.533208    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a64a6d15ed48e12e3a5e5e58422b082cc1595fb341d7d4a99905d94f968578"
Nov 01 00:50:53.021871 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:53.021838    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:50:55.037160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.032235    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.482237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.469258    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.503156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.491995    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.503156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.491967    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.575155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.568926    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 01 00:50:55.584289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.584231    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:55.584616 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.584585    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.585569 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.585543    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:55.643989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.643949    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:55.749212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.734231    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:50:55.953292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:55.953233    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.039212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.038504    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.039212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.038976    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:50:56.043157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.040445    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.567557 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.567516    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:50:56.814975 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.608671    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:50:56.817183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.608719    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:50:56.818392 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.610021    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.819033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.611147    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.848161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.611178    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.849891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.730185    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.754227    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930659    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930722    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930756    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930787    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930821    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930854    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930888    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:50:56.952161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.930920    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:50:57.053183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:56.976228    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.053183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.003964    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:50:57.053183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.052053    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.116215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.115515    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 01 00:50:57.589160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.572254    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.728157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.728104    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:50:57.728415 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.728388    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.834966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.833232    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.834966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.833698    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:57.860252 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.859309    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.038161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.859443    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.270160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.875539    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:50:58.432549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.901374    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.432925 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.901680    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:50:58.433221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:57.962659    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:50:58.433459 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.076755    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.437416 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.077023    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.437665 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.323733    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:50:58.458319 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.458270    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.741214 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.734829    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:50:58.741214 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.734900    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:58.973172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.935737    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:50:59.008449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:58.948371    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.008449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.002235    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.289034 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.288926    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.313154 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.289647    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.314260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.289688    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 01 00:50:59.324171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.323912    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:50:59.324171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.323967    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.439589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.439545    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:50:59.440179 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.440149    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:50:59.440617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.440592    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:50:59.441008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.440985    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 01 00:50:59.441503 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.441479    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.520161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.475596    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.695215 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:50:59.576691    4493 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.071s"
Nov 01 00:50:59.764156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.668257    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:50:59.764156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.759822    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:50:59.852161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.791092    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:50:59.951216 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:50:59.921389    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.003172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.001233    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.003172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.001378    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:00.003172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.001732    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:00.003172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.003068    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.043288 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.043228    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.454159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.365589    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:51:00.488364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.469694    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.488364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.469734    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.488364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.469757    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:00.554163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.501298    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.720775 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.696224    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:00.803178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.801894    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:51:00.813177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.813109    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:00.813519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:00.813448    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.002266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.002070    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.002557 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.002155    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:01.003437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.003391    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.488659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.488619    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.526408 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.489511    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.526408 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.489540    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.526408 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.490982    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.683590 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.683535    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:51:01.705312 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.705255    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.721764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.721725    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:01.721959 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.721800    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.733177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.722282    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:01.733177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.722353    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.854008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.853906    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:01.854729 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:01.854705    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.011492    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.011942    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.013100    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.026246    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.026326    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.044562    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:02.054480 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.044641    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.147362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.147236    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:02.224199 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.162554    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:51:02.310848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.164092    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.883332 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.651342    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.893339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.651387    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.895324 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.849942    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:51:02.899175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.868233    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:02.918757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:02.918714    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:51:03.157540 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.157419    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:03.161460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.161405    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:03.162598 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.162484    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:03.171183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.162964    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:03.351275 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.351221    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:03.367158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.353220    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:51:03.367158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.353291    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:51:03.367158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.353340    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:51:03.367158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.353373    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:51:03.367158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.353418    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:51:03.381283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.380706    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:51:03.382399 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.382376    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:51:03.405526 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.405496    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:51:03.423472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:03.422538    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:51:15.132895 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:51:15.132690    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:51:18.296025 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.295988    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:51:18.296614 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.296600    4493 cpu_manager.go:414] "containerMap: removing container" podUID="19d0c42a-95f8-49e0-b74e-5658509a1c82" containerName="revoke-keys"
Nov 01 00:51:18.296660 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.296615    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d0c42a-95f8-49e0-b74e-5658509a1c82" containerName="revoke-keys"
Nov 01 00:51:18.296748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.296738    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d0c42a-95f8-49e0-b74e-5658509a1c82" containerName="revoke-keys"
Nov 01 00:51:18.301528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.301512    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432356    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432591 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432492    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432591 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432567    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432604    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432648    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432778 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432816 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432797    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432874 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432819    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6np\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.432931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.432890    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.512358 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.512323    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:51:18.538002 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.537962    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.538460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.538431    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.538554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.538485    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.538554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.538527    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6np\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.538652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.538626    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539206    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539341    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539666 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539384    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539733    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539771    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.539939 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.539886    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.540108 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.540089    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.540281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.540094    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.541326 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.541305    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.542266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.542245    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.603857 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.603786    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.604523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.604499    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6np\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:18.616069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:18.616045    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:19.055371 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.055341    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:51:19.056086 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:51:19.056058    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82d413_4b6c_4871_8dc6_dfef0a86abfb.slice/crio-42186937289cba1ed0f20d3b5055e3b3539200d014869f068632ffe4d14b7a38 WatchSource:0}: Error finding container 42186937289cba1ed0f20d3b5055e3b3539200d014869f068632ffe4d14b7a38: Status 404 returned error can't find the container with id 42186937289cba1ed0f20d3b5055e3b3539200d014869f068632ffe4d14b7a38
Nov 01 00:51:19.058683 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.058648    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:19.058819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.058725    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:19.058819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.058767    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:19.351430 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.351348    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" event={"ID":"bf82d413-4b6c-4871-8dc6-dfef0a86abfb","Type":"ContainerStarted","Data":"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"}
Nov 01 00:51:19.351430 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.351406    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" event={"ID":"bf82d413-4b6c-4871-8dc6-dfef0a86abfb","Type":"ContainerStarted","Data":"42186937289cba1ed0f20d3b5055e3b3539200d014869f068632ffe4d14b7a38"}
Nov 01 00:51:19.473769 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.473713    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podStartSLOduration=1.473699127 podStartE2EDuration="1.473699127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:19.470888563 +0000 UTC m=+1669.148475513" watchObservedRunningTime="2026-11-01 00:51:19.473699127 +0000 UTC m=+1669.151286072"
Nov 01 00:51:19.626218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:19.619234    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:20.641848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:20.641812    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:20.642385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:20.641866    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:21.641642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:21.641599    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:21.641860 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:21.641669    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:22.043489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.043453    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:51:22.048684 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.048663    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.149900 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.149855    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:51:22.214966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.214936    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.214979    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215000    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wb5\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215327 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215111    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215327 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215261    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215491 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215446    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215547 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215530    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.215601 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.215582    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.326217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.326103    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.326481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.326457    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.326669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.326650    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.326817 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.326799    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.337239 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.337206    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wb5\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.337444 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.337425    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.337608 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.337591    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.337763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.330583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.337921 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.337802    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.338198 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.338180    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.338535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.338509    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.338617 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.330817    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.338807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.337459    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.339185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.339166    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.343161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.343108    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.343393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.343369    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.365159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.365105    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 00:51:22.369512 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.369493    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.398763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.398735    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wb5\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.399622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.399606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.450059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.450022    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 00:51:22.546788 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.546748    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.546876    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbt4\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.546936    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547000    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547051    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547088    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547168    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.547333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.547318    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.640457 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.640357    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:22.640649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.640590    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:22.648467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648435    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbt4\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648509    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648574    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648623    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648833 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648661    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648833 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648724    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648841    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.648941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648894    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.648951    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.649038    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.649124    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649267 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.649244    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649367 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.649350    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.649653 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.649442    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.651313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.651293    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.651453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.651436    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.680156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.672492    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:22.737464 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.732409    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.944392 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.911631    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbt4\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:22.988517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:22.988479    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:23.766734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.757297    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:51:23.768074 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.768014    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:23.817444 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.817405    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:23.817634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.817485    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:23.887993 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.886754    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f828d\" (UniqueName: \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:23.887993 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.886821    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:23.936970 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.936866    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8a74f100-dockercfg-sh79k\""
Nov 01 00:51:23.939434 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.939413    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8a74f100\""
Nov 01 00:51:23.973438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.973407    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:51:23.994440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.994396    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f828d\" (UniqueName: \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:23.994579 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.994475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:23.997918 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:23.997550    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:24.057095 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.057008    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f828d\" (UniqueName: \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\") pod \"payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:24.246100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.246068    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:51:24.247502 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:51:24.247475    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4530a76e_1ff2_4c6b_b8d4_5a8bed5766be.slice/crio-d9c1b52e1096e2c10bdfa8e84531e6757eac503d14c0451297c665c27a91b2d7 WatchSource:0}: Error finding container d9c1b52e1096e2c10bdfa8e84531e6757eac503d14c0451297c665c27a91b2d7: Status 404 returned error can't find the container with id d9c1b52e1096e2c10bdfa8e84531e6757eac503d14c0451297c665c27a91b2d7
Nov 01 00:51:24.250311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.250278    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.250420 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.250346    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.250420 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.250373    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.296104 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.296080    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:24.348540 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.348508    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 00:51:24.364157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.364104    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.364313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.364200    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.364313 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.364236    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:51:24.635162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.634236    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:24.635162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.634305    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:24.687050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.687014    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" event={"ID":"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be","Type":"ContainerStarted","Data":"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"}
Nov 01 00:51:24.687226 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.687097    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" event={"ID":"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be","Type":"ContainerStarted","Data":"d9c1b52e1096e2c10bdfa8e84531e6757eac503d14c0451297c665c27a91b2d7"}
Nov 01 00:51:24.693160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.691686    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" event={"ID":"bc012d25-c8fa-42ee-8e55-a05e2606cd87","Type":"ContainerStarted","Data":"bedb056c3dbceaacfb724d767f1391d9fade4bdeddd3c53ae2af19ed4ccd7dfc"}
Nov 01 00:51:24.693160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.691743    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" event={"ID":"bc012d25-c8fa-42ee-8e55-a05e2606cd87","Type":"ContainerStarted","Data":"019d2d7c3ba2ef066079116e96127b074404c2bbb3b6c3e8077d482ad5fff01b"}
Nov 01 00:51:24.752391 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:51:24.752362    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281fb5f1_3a9d_4f3d_904e_42a2e7641a6d.slice/crio-fc3e82a630867e22877b19b6c4664350e9938a823ec6064a17b05b18522439c4 WatchSource:0}: Error finding container fc3e82a630867e22877b19b6c4664350e9938a823ec6064a17b05b18522439c4: Status 404 returned error can't find the container with id fc3e82a630867e22877b19b6c4664350e9938a823ec6064a17b05b18522439c4
Nov 01 00:51:24.752929 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.752907    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:51:24.809830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.809779    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podStartSLOduration=3.8097640090000002 podStartE2EDuration="3.809764009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:24.808186777 +0000 UTC m=+1674.485773725" watchObservedRunningTime="2026-11-01 00:51:24.809764009 +0000 UTC m=+1674.487350953"
Nov 01 00:51:24.989020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:24.988979    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:25.622163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.621606    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:25.622163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.621664    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:25.681157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.680192    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:25.681977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.681949    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:25.682184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.682161    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:25.705394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.705351    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" event={"ID":"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d","Type":"ContainerStarted","Data":"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"}
Nov 01 00:51:25.705590 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.705412    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" event={"ID":"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d","Type":"ContainerStarted","Data":"fc3e82a630867e22877b19b6c4664350e9938a823ec6064a17b05b18522439c4"}
Nov 01 00:51:25.705667 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.705655    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:25.865156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.864946    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" podStartSLOduration=2.864925228 podStartE2EDuration="2.864925228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:25.836607805 +0000 UTC m=+1675.514194751" watchObservedRunningTime="2026-11-01 00:51:25.864925228 +0000 UTC m=+1675.542512174"
Nov 01 00:51:25.875831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.875478    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podStartSLOduration=3.875459841 podStartE2EDuration="3.875459841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:24.940119302 +0000 UTC m=+1674.617706248" watchObservedRunningTime="2026-11-01 00:51:25.875459841 +0000 UTC m=+1675.553046786"
Nov 01 00:51:25.996171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.994380    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:25.996171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:25.994443    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:26.620169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.619529    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 00:51:26.620169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.619592    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 00:51:26.675612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.675571    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:26.675807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.675635    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:26.789537 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.789504    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:51:26.790006 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.789975    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy" containerID="cri-o://31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392" gracePeriod=30
Nov 01 00:51:26.990183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.989979    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:26.990183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:26.990035    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:27.679215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.677780    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:27.679215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.677840    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:27.760159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.746208    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:27.765345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.765259    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:27.780561 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.780530    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:51:27.989500 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.989461    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:27.989710 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:27.989523    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:28.674773 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:28.674730    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:28.675228 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:28.674811    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:28.990053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:28.990018    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:28.990262 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:28.990075    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:29.675550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:29.675514    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:29.675967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:29.675575    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:29.990281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:29.990243    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:29.990462 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:29.990305    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:30.675477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.675440    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 00:51:30.675763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.675733    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 00:51:30.686093 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.686063    4493 scope.go:122] "RemoveContainer" containerID="305b34efac5a44bb5e68fc252abfc9121968bd503454472db4083b8cb68adf86"
Nov 01 00:51:30.699012 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.698984    4493 scope.go:122] "RemoveContainer" containerID="8cdff806c44e4cff8e0639a62da8505476450e27ce894dc9dd8a1f549b36cf5b"
Nov 01 00:51:30.709197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.709177    4493 scope.go:122] "RemoveContainer" containerID="be0f4183f151ad91eb68f9b327bb45b570c1e761359b3de28226009744527fbc"
Nov 01 00:51:30.989288 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.989252    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 00:51:30.989492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:30.989316    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 00:51:32.199186 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.199150    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:32.200878 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.200855    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:32.202222 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.202195    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:51:32.209871 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.209844    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:51:32.333967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.333937    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334104    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334169    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334220    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334265    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334302    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334352    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334441    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:51:32.334575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334517    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:51:32.334677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334649    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\") pod \"10b3d870-d526-42eb-bcd7-fab222214a6c\" (UID: \"10b3d870-d526-42eb-bcd7-fab222214a6c\") "
Nov 01 00:51:32.334761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:51:32.334838 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.334823    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:51:32.335072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:51:32.335858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335773    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.336125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335892    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b3d870-d526-42eb-bcd7-fab222214a6c-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.336125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335909    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.336125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335923    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.336125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.335937    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.337965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.337930    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:51:32.337965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.337938    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:51:32.338123 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.338098    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:51:32.338222 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.338162    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x" pod "10b3d870-d526-42eb-bcd7-fab222214a6c" (UID: "10b3d870-d526-42eb-bcd7-fab222214a6c"). InnerVolumeSpecName "kube-api-access-k9d9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:51:32.408286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.408228    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:32.409712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.409689    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:32.410943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.410920    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 00:51:32.437467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.437340    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d9x\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-kube-api-access-k9d9x\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.437467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.437366    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.437467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.437380    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.437467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.437395    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b3d870-d526-42eb-bcd7-fab222214a6c-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:51:32.823754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.823721    4493 generic.go:356] "Generic (PLEG): container finished" podID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerID="31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392" exitCode=0
Nov 01 00:51:32.823932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.823769    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"
Nov 01 00:51:32.823932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.823807    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" event={"ID":"10b3d870-d526-42eb-bcd7-fab222214a6c","Type":"ContainerDied","Data":"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"}
Nov 01 00:51:32.823932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.823866    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447" event={"ID":"10b3d870-d526-42eb-bcd7-fab222214a6c","Type":"ContainerDied","Data":"dc55e689eb35171f2eb2ef3b25790d8094dd311d4d047ac72872f4945faea537"}
Nov 01 00:51:32.823932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.823896    4493 scope.go:122] "RemoveContainer" containerID="31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"
Nov 01 00:51:32.834440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.834414    4493 scope.go:122] "RemoveContainer" containerID="31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"
Nov 01 00:51:32.834778 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:51:32.834754    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392\": container with ID starting with 31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392 not found: ID does not exist" containerID="31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"
Nov 01 00:51:32.834835 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:32.834789    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392"} err="failed to get container status \"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392\": rpc error: code = NotFound desc = could not find container \"31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392\": container with ID starting with 31f5a713a3b92a2f3ef0d3eecac8f593d77f78e13fcba639e60133dd90bed392 not found: ID does not exist"
Nov 01 00:51:33.010376 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:33.010344    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:51:33.014121 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:33.014098    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cs447"]
Nov 01 00:51:34.504632 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:34.504603    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" path="/var/lib/kubelet/pods/10b3d870-d526-42eb-bcd7-fab222214a6c/volumes"
Nov 01 00:51:36.756390 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:36.756344    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:51:38.490163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:38.489252    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:38.836907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:38.836710    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.017343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.017279    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 01 00:51:39.017702 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.017673    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.019271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.019231    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.067324 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.067265    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:39.117334 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.117238    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:39.356167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.328171    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:39.472247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.390617    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:39.472247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.450023    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.472247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.450238    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:39.472247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.450263    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.858218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.845472    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:39.858218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.847319    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:51:39.934193 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:39.920679    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Nov 01 00:51:40.089161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.067539    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:40.211221 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.090052    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.226269 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.091553    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.244150 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.244092    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:51:40.297173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.130476    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 01 00:51:40.297173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.131862    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.338184 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.336529    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:51:40.399610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.399578    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:51:40.473700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.473657    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:40.504356 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.504304    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:51:40.526163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.506723    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:40.552794 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.526647    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:40.558345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.558313    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:51:40.567092 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.526704    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.568403 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.526725    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.569727 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.569682    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:51:40.569897 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.569883    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:51:40.590159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.526682    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:51:40.592747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:40.592715    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:51:41.204302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.204247    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.239975 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.227348    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.244954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.244917    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:41.245257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.245224    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:41.273159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.254655    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.417427 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.417379    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:41.498842 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.498789    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:41.592161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.576083    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.592161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.576270    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:41.592161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.576553    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.592161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.576838    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 01 00:51:41.592161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.577242    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:51:41.614202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:41.602256    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:42.399160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.393358    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:42.399160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.394600    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.413158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.403641    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.603162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.589052    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.603162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.592993    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.609164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.608078    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.613265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.613227    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:42.620492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.618437    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:42.620492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.618788    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:51:42.620796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:42.620768    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:51:43.012157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.009530    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:43.012157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.009734    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.223158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.215633    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:43.223158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.215721    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.430254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.399534    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 01 00:51:43.430254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.400481    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.430254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.406378    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.593771 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.593719    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:51:43.594539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.594505    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.636705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.636650    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.650359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.650319    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:51:43.658783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.658728    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:43.671862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.671820    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.767146 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.758293    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.931175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.842475    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:43.931175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.842552    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:43.931175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.851878    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:43.931175 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:43.851956    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.420333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.419275    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.420333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.419692    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.420333 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.420024    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:44.473711 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.473668    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.599230 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.597761    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.600200 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.600148    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:51:44.656054 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.656008    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.661528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.661479    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:51:44.667300 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.667264    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:44.711277 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.702329    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:44.766385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:44.766341    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.428510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.428452    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.430750 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.428809    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 00:51:45.431473 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.431443    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.527636 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.527593    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.675909 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.675854    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.833573 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.688302    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.841161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.688426    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.857711 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.739327    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.739368    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.771730    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.773839    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.773973    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.861998    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.846662    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.863012    4493 cpu_manager.go:414] "containerMap: removing container" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.863032    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy"
Nov 01 00:51:45.865320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.863307    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b3d870-d526-42eb-bcd7-fab222214a6c" containerName="istio-proxy"
Nov 01 00:51:45.931668 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:51:45.931461    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:51:45.975080 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:45.935490    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.006635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.003600    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-14957a74-dockercfg-j6ncl\""
Nov 01 00:51:46.006635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.004214    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-14957a74\""
Nov 01 00:51:46.035773 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.035672    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 01 00:51:46.195169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.195016    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:51:46.233185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.220562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.233185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.220727    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms78p\" (UniqueName: \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.303110 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.303073    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:51:46.331438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.330835    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.331438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.330987    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms78p\" (UniqueName: \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.332233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.332201    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.332643 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.332584    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.440633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.440596    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.449314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.449230    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwznc\" (UniqueName: \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.449902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.449868    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:51:46.450111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.450087    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.450971 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.450944    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.460252 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.460214    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.475118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.475083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms78p\" (UniqueName: \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\") pod \"payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.553109 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.553076    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.562166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.557256    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwznc\" (UniqueName: \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.562447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.560224    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.588723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.588412    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:46.820153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.802409    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwznc\" (UniqueName: \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\") pod \"payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") " pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:46.820153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.814499    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:51:46.844151 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.843220    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.862319 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.862275    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.871166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.868718    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:51:46.871166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.868965    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:51:46.873442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.871565    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:51:46.890158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.884248    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:51:46.927358 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:46.927328    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:51:47.025379 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.022845    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:51:47.036639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.036613    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:47.038520 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.038501    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:51:47.084081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.084052    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:51:47.128161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.115465    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:51:47.130311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.130286    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:51:47.198892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.198865    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:51:47.208692 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.208665    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:51:47.330895 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.329865    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:51:47.733957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.733926    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:51:47.928235 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:51:47.926693    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode623ec1e_a1df_40f2_9ed3_2855aafbd948.slice/crio-67debebf862b8651f4da2997765940d356f27ea75bf783ccef122debfa5fda0e WatchSource:0}: Error finding container 67debebf862b8651f4da2997765940d356f27ea75bf783ccef122debfa5fda0e: Status 404 returned error can't find the container with id 67debebf862b8651f4da2997765940d356f27ea75bf783ccef122debfa5fda0e
Nov 01 00:51:47.928235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:47.926726    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:51:48.522921 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.522887    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" event={"ID":"2a4f1b5c-55ff-400d-9012-97464be1eca2","Type":"ContainerStarted","Data":"54e3c16ed87ba47027bd17b42e32796751e40d873149f67c532867b39e45a154"}
Nov 01 00:51:48.523173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.522940    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" event={"ID":"2a4f1b5c-55ff-400d-9012-97464be1eca2","Type":"ContainerStarted","Data":"7ffb158edfeb88ff8065594b67ffa93f00336ed27c04dbb5de280a5abddcb719"}
Nov 01 00:51:48.523173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.522962    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:48.524805 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.524776    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" event={"ID":"e623ec1e-a1df-40f2-9ed3-2855aafbd948","Type":"ContainerStarted","Data":"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"}
Nov 01 00:51:48.524940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.524821    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" event={"ID":"e623ec1e-a1df-40f2-9ed3-2855aafbd948","Type":"ContainerStarted","Data":"67debebf862b8651f4da2997765940d356f27ea75bf783ccef122debfa5fda0e"}
Nov 01 00:51:48.524940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.524870    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:48.643840 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:48.643784    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" podStartSLOduration=3.643766227 podStartE2EDuration="3.643766227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:48.638342753 +0000 UTC m=+1698.315929690" watchObservedRunningTime="2026-11-01 00:51:48.643766227 +0000 UTC m=+1698.321353175"
Nov 01 00:51:59.535381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.535349    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:51:59.535868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.535422    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:51:59.540526 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.540482    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" podStartSLOduration=14.540468959 podStartE2EDuration="14.540468959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:48.762617662 +0000 UTC m=+1698.440204608" watchObservedRunningTime="2026-11-01 00:51:59.540468959 +0000 UTC m=+1709.218055906"
Nov 01 00:51:59.541231 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.541212    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:51:59.545854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.545835    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.631845 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.631812    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb57n\" (UniqueName: \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.632237 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.632209    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.632445 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.632426    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.632799 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.632769    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.638385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.638361    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-64566096-serving-cert\""
Nov 01 00:51:59.642487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.642466    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.735754    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb57n\" (UniqueName: \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.735848    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.736013    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:51:59.736116    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-64566096-serving-cert: secret "maas-api-e2e-ait-64566096-serving-cert" not found
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.736157    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:51:59.736219    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls podName:7b0b3c3c-65d3-4ea6-82d8-29716e17422e nodeName:}" failed. No retries permitted until 2026-11-01 00:52:00.236204571 +0000 UTC m=+1709.913791510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls") pod "maas-api-e2e-ait-64566096-6896b56889-q9prt" (UID: "7b0b3c3c-65d3-4ea6-82d8-29716e17422e") : secret "maas-api-e2e-ait-64566096-serving-cert" not found
Nov 01 00:51:59.737182 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.736949    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.738896 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.738867    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:51:59.805144 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:51:59.805052    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb57n\" (UniqueName: \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:00.250997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:00.250959    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:00.253626 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:00.253585    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") pod \"maas-api-e2e-ait-64566096-6896b56889-q9prt\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:00.461519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:00.461485    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:00.913838 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:00.913810    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:52:00.914746 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:52:00.914722    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0b3c3c_65d3_4ea6_82d8_29716e17422e.slice/crio-31fda932771c635d6fe0a3c86ccb833bff4a9e2ad3c3a651d5c313067ae9f17c WatchSource:0}: Error finding container 31fda932771c635d6fe0a3c86ccb833bff4a9e2ad3c3a651d5c313067ae9f17c: Status 404 returned error can't find the container with id 31fda932771c635d6fe0a3c86ccb833bff4a9e2ad3c3a651d5c313067ae9f17c
Nov 01 00:52:00.916316 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:00.916293    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:52:01.633987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:01.633950    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" event={"ID":"7b0b3c3c-65d3-4ea6-82d8-29716e17422e","Type":"ContainerStarted","Data":"31fda932771c635d6fe0a3c86ccb833bff4a9e2ad3c3a651d5c313067ae9f17c"}
Nov 01 00:52:02.698156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:02.694788    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" event={"ID":"7b0b3c3c-65d3-4ea6-82d8-29716e17422e","Type":"ContainerStarted","Data":"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"}
Nov 01 00:52:02.698156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:02.695909    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:04.619188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.619122    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" podStartSLOduration=4.65952148 podStartE2EDuration="5.619109089s" totalImagesPullingTime="959.587609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:02.833479526 +0000 UTC m=+1712.511066471" watchObservedRunningTime="2026-11-01 00:52:04.619109089 +0000 UTC m=+1714.296696034"
Nov 01 00:52:04.622770 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.622748    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:52:04.627317 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.627298    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.679655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.679628    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8a74f100-serving-cert\""
Nov 01 00:52:04.734381 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.734347    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:52:04.742941 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.742910    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.743189 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.743165    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.743281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.743219    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.743359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.743320    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpnt\" (UniqueName: \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.845339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.845296    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.845541 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.845502    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.845541 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.845531    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.845682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.845615    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpnt\" (UniqueName: \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.846098 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.846073    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.848107 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.848083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.848238 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.848222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.906983 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.906911    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpnt\" (UniqueName: \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\") pod \"maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:04.941067 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:04.941038    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:05.370059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:05.370025    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:52:05.370553 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:52:05.370524    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd9532_d7fe_433b_9e71_e07a1628b2c9.slice/crio-8a0b2947bf5d22b8caf1782b805bb31fb1b2e15627d672163a9c190a5544115a WatchSource:0}: Error finding container 8a0b2947bf5d22b8caf1782b805bb31fb1b2e15627d672163a9c190a5544115a: Status 404 returned error can't find the container with id 8a0b2947bf5d22b8caf1782b805bb31fb1b2e15627d672163a9c190a5544115a
Nov 01 00:52:05.719123 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:05.719091    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" event={"ID":"71bd9532-d7fe-433b-9e71-e07a1628b2c9","Type":"ContainerStarted","Data":"8a0b2947bf5d22b8caf1782b805bb31fb1b2e15627d672163a9c190a5544115a"}
Nov 01 00:52:06.806916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:06.806879    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" event={"ID":"71bd9532-d7fe-433b-9e71-e07a1628b2c9","Type":"ContainerStarted","Data":"3d177cbd06777d7bc40f8e4ef505acda893a3573ef094a39ba48a4d5f5bd3c33"}
Nov 01 00:52:06.809266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:06.807850    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:06.928020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:06.927972    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" podStartSLOduration=2.000506862 podStartE2EDuration="2.927957797s" totalImagesPullingTime="927.450935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:06.925569816 +0000 UTC m=+1716.603156762" watchObservedRunningTime="2026-11-01 00:52:06.927957797 +0000 UTC m=+1716.605544741"
Nov 01 00:52:09.716185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:09.716125    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:52:12.166515 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.166467    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 00:52:12.173519 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.173492    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209448    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209619 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209484    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209619 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209605    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsv8\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209693 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209661    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209773 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209761    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209808 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209784    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209808 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209803    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209883 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209827    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.209883 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.209854    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.213243 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.213220    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 00:52:12.217440 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.217416    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.267522 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.267488    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 00:52:12.311774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311519    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.311774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311644    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.311774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311675    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.311774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311708    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.311774 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311733    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.312331 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311825    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsv8\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.312331 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311900    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.312331 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311943    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.312499 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.312439    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.311996    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.313123 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313096    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313229 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313229 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313216    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.313343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313243    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313292    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.313343 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313330    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvwx\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.313548 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313383    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.313548 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.313461    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.316027 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.316002    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.316380 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.316348    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.316495 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.316462    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.316611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.316594    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.316868 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.316846    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.317172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.317122    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.321295 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.321269    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 00:52:12.372917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.372891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.373271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.373251    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsv8\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\") pod \"maas-default-gateway-openshift-default-67bf986f99-qplbs\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.414728 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.414692    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.414728 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.414736    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.414986 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.414835    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.414986 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.414868    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415013    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415060    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvwx\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415109    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415240 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415215    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415317    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415668 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415730    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.415883 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.415862    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.416035 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.416015    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.416096 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.416034    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.417512 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.417453    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.418372 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.418347    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.476113 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.476078    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.476272 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.476224    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvwx\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\") pod \"maas-default-gateway-openshift-default-67bf986f99-rvm88\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.489018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.488993    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:12.531664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.531631    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:12.961890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.961870    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 00:52:12.962041 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:52:12.962023    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7df77c_4996_41de_b23d_020634c40e77.slice/crio-cb14b66b2887d85bc7158ac0112fa86fdd77b440413b88092aed5b3847e4f3c4 WatchSource:0}: Error finding container cb14b66b2887d85bc7158ac0112fa86fdd77b440413b88092aed5b3847e4f3c4: Status 404 returned error can't find the container with id cb14b66b2887d85bc7158ac0112fa86fdd77b440413b88092aed5b3847e4f3c4
Nov 01 00:52:12.964520 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:52:12.964498    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5f909a_afc3_4541_9c91_2d692f814a37.slice/crio-bb795282843ce483019af27fc35df58df313e3cf2e04d042433431cb8e8cb78f WatchSource:0}: Error finding container bb795282843ce483019af27fc35df58df313e3cf2e04d042433431cb8e8cb78f: Status 404 returned error can't find the container with id bb795282843ce483019af27fc35df58df313e3cf2e04d042433431cb8e8cb78f
Nov 01 00:52:12.965015 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.964982    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:12.965087 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.965072    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:12.965126 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.965117    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:12.966744 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.966723    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 00:52:12.967301 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.967276    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:12.967353 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.967329    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:12.967396 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:12.967363    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:52:13.844847 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:13.844820    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:52:13.953158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:13.952975    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" event={"ID":"2c7df77c-4996-41de-b23d-020634c40e77","Type":"ContainerStarted","Data":"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"}
Nov 01 00:52:13.953158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:13.953047    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" event={"ID":"2c7df77c-4996-41de-b23d-020634c40e77","Type":"ContainerStarted","Data":"cb14b66b2887d85bc7158ac0112fa86fdd77b440413b88092aed5b3847e4f3c4"}
Nov 01 00:52:13.967283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:13.961178    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" event={"ID":"3e5f909a-afc3-4541-9c91-2d692f814a37","Type":"ContainerStarted","Data":"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"}
Nov 01 00:52:13.967283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:13.961258    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" event={"ID":"3e5f909a-afc3-4541-9c91-2d692f814a37","Type":"ContainerStarted","Data":"bb795282843ce483019af27fc35df58df313e3cf2e04d042433431cb8e8cb78f"}
Nov 01 00:52:14.172613 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.172092    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podStartSLOduration=2.172067984 podStartE2EDuration="2.172067984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:14.12777946 +0000 UTC m=+1723.805366404" watchObservedRunningTime="2026-11-01 00:52:14.172067984 +0000 UTC m=+1723.849654930"
Nov 01 00:52:14.273458 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.273405    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podStartSLOduration=2.273388121 podStartE2EDuration="2.273388121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:14.266820107 +0000 UTC m=+1723.944407052" watchObservedRunningTime="2026-11-01 00:52:14.273388121 +0000 UTC m=+1723.950975065"
Nov 01 00:52:14.489985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.489957    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:14.491493 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.491468    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:14.491622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.491517    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:14.539159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.538426    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:14.565157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.564246    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:14.565157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:14.564310    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:15.185707 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:52:15.185675    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:52:15.491621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:15.490468    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:15.491621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:15.490539    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:15.538172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:15.537597    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:15.538172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:15.537658    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:16.489912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:16.489875    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:16.490320 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:16.489933    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:16.537160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:16.532464    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:16.537160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:16.532526    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:17.489747 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:17.489713    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:17.489929 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:17.489764    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:17.538159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:17.537582    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:17.538159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:17.537645    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:18.489477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:18.489445    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:18.489651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:18.489496    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:18.543158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:18.542401    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:18.543158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:18.542477    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:19.462042 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:19.458668    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:52:19.510559 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:19.510524    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:19.510831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:19.510806    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:19.532961 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:19.532928    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:19.533255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:19.533224    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:20.491547 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:20.491507    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:20.492195 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:20.492104    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:20.536177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:20.532555    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:20.536177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:20.532614    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:21.489751 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:21.489717    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:21.490033 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:21.490006    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:21.532663 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:21.532627    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:21.533094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:21.532736    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:22.490373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:22.490333    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 01 00:52:22.490978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:22.490431    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 01 00:52:22.532624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:22.532586    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 01 00:52:22.532949 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:22.532904    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 01 00:52:24.286977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.282759    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:24.486274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.486229    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:24.504048 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.504003    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:24.534155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.531324    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qplbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:52:24.534155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.531404    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:24.536334 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.536289    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:52:24.536626 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.536600    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:24.724256 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.723557    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:24.725146 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.725094    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:24.729219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:24.729192    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 00:52:25.544397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:25.544356    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:52:25.545018 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:25.544983    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:26.587339 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:26.587301    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rvm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:52:26.587830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:26.587384    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:27.238889 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:27.238852    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:27.239514 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:27.239485    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:27.240476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:27.240455    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 00:52:44.991105 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:52:44.991016    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:52:57.029685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:57.029648    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:52:59.417291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.417243    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.469014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.468248    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.534173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.529246    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.581241 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.581192    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.586985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.586944    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.634890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.633692    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.668215 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.664238    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.782158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.781437    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:52:59.947209 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.933216    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Nov 01 00:52:59.979157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:52:59.978223    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.034159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.033659    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 01 00:53:00.262286 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.262242    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.287302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.287251    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.476449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.476384    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.477869 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.477848    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:53:00.588212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.585224    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 00:53:00.588212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.585954    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:53:00.588212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.587405    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:53:00.588212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.587711    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:53:00.640181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.633874    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.640181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.634246    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:53:00.665266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.665219    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 01 00:53:00.665908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.665881    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:53:00.812160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.796658    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:00.812160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:00.797071    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:53:01.323753 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.323450    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:01.352193 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.352162    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:53:01.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.411279    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:01.428728 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.427265    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 00:53:01.477649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.477610    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 00:53:01.555166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.532519    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:53:01.705588 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.634758    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:53:01.706037 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.650034    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:01.706362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.677243    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:01.741670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.741626    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 00:53:01.797162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:01.797089    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:02.001951 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.001369    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 00:53:02.143155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.139451    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 00:53:02.338225 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.337239    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:02.355233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.338468    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:02.424158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.414436    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 00:53:02.450160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.446334    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:53:02.753945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.753894    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 01 00:53:02.830161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:02.803337    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:53:03.048217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.033283    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:03.271159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.252273    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:53:03.271159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.252336    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 00:53:03.271159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.252377    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 00:53:03.344274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.339962    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:03.344274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.340874    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:53:03.419378 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.419313    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:03.493085 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.493055    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 00:53:03.641705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.561222    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 01 00:53:03.796168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.794431    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:53:03.884805 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.884749    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 00:53:03.912161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.912009    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 00:53:03.937987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:03.937772    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 00:53:14.998701 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:53:14.998666    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:53:24.273754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:24.273724    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:53:29.274067 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.274011    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" podUID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" containerName="payload-pre-processing" containerID="cri-o://df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a" gracePeriod=25
Nov 01 00:53:29.562605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.562580    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:53:29.678699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.678655    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\") pod \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") "
Nov 01 00:53:29.678994 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.678973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\" (UniqueName: \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\") pod \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\" (UID: \"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d\") "
Nov 01 00:53:29.679212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.679183    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume" pod "281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" (UID: "281fb5f1-3a9d-4f3d-904e-42a2e7641a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:53:29.679699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.679673    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:29.681670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.681645    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d" pod "281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" (UID: "281fb5f1-3a9d-4f3d-904e-42a2e7641a6d"). InnerVolumeSpecName "kube-api-access-f828d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:53:29.781615 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:29.781583    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f828d\" (UniqueName: \"kubernetes.io/projected/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d-kube-api-access-f828d\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:30.228212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.228178    4493 generic.go:356] "Generic (PLEG): container finished" podID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" containerID="df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a" exitCode=0
Nov 01 00:53:30.228395 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.228224    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"
Nov 01 00:53:30.228395 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.228256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" event={"ID":"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d","Type":"ContainerDied","Data":"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"}
Nov 01 00:53:30.228395 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.228312    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn" event={"ID":"281fb5f1-3a9d-4f3d-904e-42a2e7641a6d","Type":"ContainerDied","Data":"fc3e82a630867e22877b19b6c4664350e9938a823ec6064a17b05b18522439c4"}
Nov 01 00:53:30.228395 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.228333    4493 scope.go:122] "RemoveContainer" containerID="df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"
Nov 01 00:53:30.239661 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.239639    4493 scope.go:122] "RemoveContainer" containerID="df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"
Nov 01 00:53:30.239957 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:53:30.239937    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a\": container with ID starting with df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a not found: ID does not exist" containerID="df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"
Nov 01 00:53:30.240007 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.239966    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a"} err="failed to get container status \"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a\": rpc error: code = NotFound desc = could not find container \"df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a\": container with ID starting with df4f5515bfccb7896c5722549694a533e869430a3088fb330c0ec7855f150e2a not found: ID does not exist"
Nov 01 00:53:30.394968 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.394932    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:53:30.445349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.445316    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a74f100-84b4fc978c-8cvbn"]
Nov 01 00:53:30.504616 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:30.504547    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" path="/var/lib/kubelet/pods/281fb5f1-3a9d-4f3d-904e-42a2e7641a6d/volumes"
Nov 01 00:53:41.293997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:41.293960    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:53:45.090169 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:53:45.088509    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:53:51.811862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:51.811825    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:53:51.812860 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:51.812763    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" podUID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" containerName="maas-api" containerID="cri-o://9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca" gracePeriod=30
Nov 01 00:53:52.424167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.418296    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:53:52.544094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.543981    4493 generic.go:356] "Generic (PLEG): container finished" podID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" containerID="9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca" exitCode=0
Nov 01 00:53:52.544369 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.544348    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"
Nov 01 00:53:52.547916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.547881    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" event={"ID":"7b0b3c3c-65d3-4ea6-82d8-29716e17422e","Type":"ContainerDied","Data":"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"}
Nov 01 00:53:52.548072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.547940    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt" event={"ID":"7b0b3c3c-65d3-4ea6-82d8-29716e17422e","Type":"ContainerDied","Data":"31fda932771c635d6fe0a3c86ccb833bff4a9e2ad3c3a651d5c313067ae9f17c"}
Nov 01 00:53:52.548072 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.547964    4493 scope.go:122] "RemoveContainer" containerID="9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"
Nov 01 00:53:52.560197 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.560177    4493 scope.go:122] "RemoveContainer" containerID="9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"
Nov 01 00:53:52.560500 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:53:52.560473    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca\": container with ID starting with 9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca not found: ID does not exist" containerID="9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"
Nov 01 00:53:52.560551 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.560510    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca"} err="failed to get container status \"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca\": rpc error: code = NotFound desc = could not find container \"9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca\": container with ID starting with 9dab111f10857e9f8fd8d52aff6554e56dcdec760a6485a7a4345cfa614e73ca not found: ID does not exist"
Nov 01 00:53:52.570977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.570943    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") pod \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") "
Nov 01 00:53:52.571112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.571010    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\") pod \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") "
Nov 01 00:53:52.571112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.571044    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\") pod \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") "
Nov 01 00:53:52.571371 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.571349    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\" (UniqueName: \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\") pod \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\" (UID: \"7b0b3c3c-65d3-4ea6-82d8-29716e17422e\") "
Nov 01 00:53:52.572024 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.571999    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca" pod "7b0b3c3c-65d3-4ea6-82d8-29716e17422e" (UID: "7b0b3c3c-65d3-4ea6-82d8-29716e17422e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:53:52.573900 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.573873    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls" pod "7b0b3c3c-65d3-4ea6-82d8-29716e17422e" (UID: "7b0b3c3c-65d3-4ea6-82d8-29716e17422e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:53:52.573978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.573894    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs" pod "7b0b3c3c-65d3-4ea6-82d8-29716e17422e" (UID: "7b0b3c3c-65d3-4ea6-82d8-29716e17422e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:53:52.574122 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.574097    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n" pod "7b0b3c3c-65d3-4ea6-82d8-29716e17422e" (UID: "7b0b3c3c-65d3-4ea6-82d8-29716e17422e"). InnerVolumeSpecName "kube-api-access-qb57n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:53:52.673972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.673938    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb57n\" (UniqueName: \"kubernetes.io/projected/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-kube-api-access-qb57n\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:52.673972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.673970    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:52.673972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.673980    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:52.674223 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:52.673988    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0b3c3c-65d3-4ea6-82d8-29716e17422e-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:53.032670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.032629    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:53:53.037651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.037589    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" podUID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" containerName="maas-api" containerID="cri-o://3d177cbd06777d7bc40f8e4ef505acda893a3573ef094a39ba48a4d5f5bd3c33" gracePeriod=30
Nov 01 00:53:53.054842 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.054810    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:53:53.073210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.072956    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64566096-6896b56889-q9prt"]
Nov 01 00:53:53.644164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.640785    4493 generic.go:356] "Generic (PLEG): container finished" podID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" containerID="3d177cbd06777d7bc40f8e4ef505acda893a3573ef094a39ba48a4d5f5bd3c33" exitCode=0
Nov 01 00:53:53.644164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.640891    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" event={"ID":"71bd9532-d7fe-433b-9e71-e07a1628b2c9","Type":"ContainerDied","Data":"3d177cbd06777d7bc40f8e4ef505acda893a3573ef094a39ba48a4d5f5bd3c33"}
Nov 01 00:53:53.729060 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.729036    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:53:53.858173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.839884    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\") pod \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") "
Nov 01 00:53:53.858173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.856875    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\") pod \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") "
Nov 01 00:53:53.858173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.857050    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\" (UniqueName: \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\") pod \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") "
Nov 01 00:53:53.858173 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.857201    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\") pod \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\" (UID: \"71bd9532-d7fe-433b-9e71-e07a1628b2c9\") "
Nov 01 00:53:53.862282 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.861678    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca" pod "71bd9532-d7fe-433b-9e71-e07a1628b2c9" (UID: "71bd9532-d7fe-433b-9e71-e07a1628b2c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:53:53.893647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.889322    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls" pod "71bd9532-d7fe-433b-9e71-e07a1628b2c9" (UID: "71bd9532-d7fe-433b-9e71-e07a1628b2c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:53:53.908039 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.908010    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs" pod "71bd9532-d7fe-433b-9e71-e07a1628b2c9" (UID: "71bd9532-d7fe-433b-9e71-e07a1628b2c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:53:53.922239 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.922205    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt" pod "71bd9532-d7fe-433b-9e71-e07a1628b2c9" (UID: "71bd9532-d7fe-433b-9e71-e07a1628b2c9"). InnerVolumeSpecName "kube-api-access-hlpnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:53:53.959487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.959457    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:53.959487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.959484    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bd9532-d7fe-433b-9e71-e07a1628b2c9-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:53.959487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.959495    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpnt\" (UniqueName: \"kubernetes.io/projected/71bd9532-d7fe-433b-9e71-e07a1628b2c9-kube-api-access-hlpnt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:53.959717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:53.959505    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bd9532-d7fe-433b-9e71-e07a1628b2c9-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:53:54.578167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.577722    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" path="/var/lib/kubelet/pods/7b0b3c3c-65d3-4ea6-82d8-29716e17422e/volumes"
Nov 01 00:53:54.699165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.697347    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf" event={"ID":"71bd9532-d7fe-433b-9e71-e07a1628b2c9","Type":"ContainerDied","Data":"8a0b2947bf5d22b8caf1782b805bb31fb1b2e15627d672163a9c190a5544115a"}
Nov 01 00:53:54.699165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.697434    4493 scope.go:122] "RemoveContainer" containerID="3d177cbd06777d7bc40f8e4ef505acda893a3573ef094a39ba48a4d5f5bd3c33"
Nov 01 00:53:54.699165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.697556    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"
Nov 01 00:53:54.885397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.885272    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:53:54.889511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:54.889491    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a74f100-5979cc49b-rb4tf"]
Nov 01 00:53:55.359066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.359032    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:53:55.475550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.475520    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:53:55.492204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492174    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:53:55.492802 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492787    4493 cpu_manager.go:414] "containerMap: removing container" podUID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" containerName="maas-api"
Nov 01 00:53:55.492802 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492802    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" containerName="maas-api"
Nov 01 00:53:55.492916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492846    4493 cpu_manager.go:414] "containerMap: removing container" podUID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" containerName="payload-pre-processing"
Nov 01 00:53:55.492916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492852    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" containerName="payload-pre-processing"
Nov 01 00:53:55.492916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492863    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" containerName="maas-api"
Nov 01 00:53:55.492916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492868    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" containerName="maas-api"
Nov 01 00:53:55.493425 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492952    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="281fb5f1-3a9d-4f3d-904e-42a2e7641a6d" containerName="payload-pre-processing"
Nov 01 00:53:55.493425 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492959    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0b3c3c-65d3-4ea6-82d8-29716e17422e" containerName="maas-api"
Nov 01 00:53:55.493425 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.492969    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" containerName="maas-api"
Nov 01 00:53:55.497945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.497927    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.559077 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.559041    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:53:55.559530 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.559464    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-13f6ab63-openshift-default-dockercfg-z5v4m\""
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709643    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwmx\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709694    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709772    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709837    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709873    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709956    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.709999    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.710114    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.715217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.710161    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827619    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827696    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827731    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827798    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827835    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827928    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.827960    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.828200    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwmx\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.828253    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.828642    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.828695    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.852789 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.828897    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.878723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.877668    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.878723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.878062    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:55.988780 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:55.921066    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:56.015203 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:56.015083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:56.052156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:56.043712    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:56.052156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:56.046618    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwmx\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\") pod \"e2e-ait-13f6ab63-openshift-default-5484496857-54zk9\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") " pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:56.115254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:56.115216    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:56.537374 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:56.535078    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bd9532-d7fe-433b-9e71-e07a1628b2c9" path="/var/lib/kubelet/pods/71bd9532-d7fe-433b-9e71-e07a1628b2c9/volumes"
Nov 01 00:53:57.231084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:57.229945    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:53:57.231084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:57.230230    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:53:57.231084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:57.230283    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:53:57.231084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:57.230927    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:53:58.007956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.004341    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerStarted","Data":"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"}
Nov 01 00:53:58.007956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.004413    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerStarted","Data":"29849ff9d2de1814856a5b1fc6c7918d8e5fb5cc0652e9f1b1b7dad2e81011e1"}
Nov 01 00:53:58.118386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.118304    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:53:58.118918 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.118631    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:53:58.118918 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.118690    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:53:58.162157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:58.158111    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podStartSLOduration=3.1580859119999998 podStartE2EDuration="3.158085912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:53:58.149269014 +0000 UTC m=+1827.826855960" watchObservedRunningTime="2026-11-01 00:53:58.158085912 +0000 UTC m=+1827.835672858"
Nov 01 00:53:59.122022 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:53:59.121990    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:54:00.019471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:00.019437    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:54:00.020522 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:00.020499    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:54:00.359848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:00.359752    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" podUID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" containerName="payload-processing" containerID="cri-o://2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5" gracePeriod=25
Nov 01 00:54:00.476682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:00.476614    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" podUID="2a4f1b5c-55ff-400d-9012-97464be1eca2" containerName="payload-pre-processing" containerID="cri-o://54e3c16ed87ba47027bd17b42e32796751e40d873149f67c532867b39e45a154" gracePeriod=25
Nov 01 00:54:00.988050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:00.988012    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:54:01.027030 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.027001    4493 generic.go:356] "Generic (PLEG): container finished" podID="2a4f1b5c-55ff-400d-9012-97464be1eca2" containerID="54e3c16ed87ba47027bd17b42e32796751e40d873149f67c532867b39e45a154" exitCode=0
Nov 01 00:54:01.027224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.027073    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" event={"ID":"2a4f1b5c-55ff-400d-9012-97464be1eca2","Type":"ContainerDied","Data":"54e3c16ed87ba47027bd17b42e32796751e40d873149f67c532867b39e45a154"}
Nov 01 00:54:01.028542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.028521    4493 generic.go:356] "Generic (PLEG): container finished" podID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" containerID="2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5" exitCode=0
Nov 01 00:54:01.028675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.028568    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"
Nov 01 00:54:01.028675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.028603    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" event={"ID":"e623ec1e-a1df-40f2-9ed3-2855aafbd948","Type":"ContainerDied","Data":"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"}
Nov 01 00:54:01.028675 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.028654    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h" event={"ID":"e623ec1e-a1df-40f2-9ed3-2855aafbd948","Type":"ContainerDied","Data":"67debebf862b8651f4da2997765940d356f27ea75bf783ccef122debfa5fda0e"}
Nov 01 00:54:01.028794 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.028684    4493 scope.go:122] "RemoveContainer" containerID="2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"
Nov 01 00:54:01.042679 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.042203    4493 scope.go:122] "RemoveContainer" containerID="2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"
Nov 01 00:54:01.042679 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:54:01.042596    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5\": container with ID starting with 2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5 not found: ID does not exist" containerID="2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"
Nov 01 00:54:01.042679 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.042633    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5"} err="failed to get container status \"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5\": rpc error: code = NotFound desc = could not find container \"2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5\": container with ID starting with 2ffa06060b173b90f6b133a6936c14dbf4ba6294962a32c03833afb3ddbd3cc5 not found: ID does not exist"
Nov 01 00:54:01.065723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.065693    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\" (UniqueName: \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\") pod \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") "
Nov 01 00:54:01.065862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.065752    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\") pod \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\" (UID: \"e623ec1e-a1df-40f2-9ed3-2855aafbd948\") "
Nov 01 00:54:01.066167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.066103    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume" pod "e623ec1e-a1df-40f2-9ed3-2855aafbd948" (UID: "e623ec1e-a1df-40f2-9ed3-2855aafbd948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:54:01.066723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.066705    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623ec1e-a1df-40f2-9ed3-2855aafbd948-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:54:01.068923 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.068015    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc" pod "e623ec1e-a1df-40f2-9ed3-2855aafbd948" (UID: "e623ec1e-a1df-40f2-9ed3-2855aafbd948"). InnerVolumeSpecName "kube-api-access-pwznc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:54:01.168265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.168234    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwznc\" (UniqueName: \"kubernetes.io/projected/e623ec1e-a1df-40f2-9ed3-2855aafbd948-kube-api-access-pwznc\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:54:01.170267 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.170202    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:54:01.268782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.268749    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\") pod \"2a4f1b5c-55ff-400d-9012-97464be1eca2\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") "
Nov 01 00:54:01.269119 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.269096    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\" (UniqueName: \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\") pod \"2a4f1b5c-55ff-400d-9012-97464be1eca2\" (UID: \"2a4f1b5c-55ff-400d-9012-97464be1eca2\") "
Nov 01 00:54:01.269191 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.269159    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume" pod "2a4f1b5c-55ff-400d-9012-97464be1eca2" (UID: "2a4f1b5c-55ff-400d-9012-97464be1eca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:54:01.269717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.269694    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4f1b5c-55ff-400d-9012-97464be1eca2-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:54:01.271346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.271321    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p" pod "2a4f1b5c-55ff-400d-9012-97464be1eca2" (UID: "2a4f1b5c-55ff-400d-9012-97464be1eca2"). InnerVolumeSpecName "kube-api-access-ms78p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:54:01.371061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.371031    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms78p\" (UniqueName: \"kubernetes.io/projected/2a4f1b5c-55ff-400d-9012-97464be1eca2-kube-api-access-ms78p\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:54:01.455889 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.455815    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:54:01.462644 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:01.462599    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-14957a74-6597f9bb4-wsj9h"]
Nov 01 00:54:02.037558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.037530    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"
Nov 01 00:54:02.037558 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.037542    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4" event={"ID":"2a4f1b5c-55ff-400d-9012-97464be1eca2","Type":"ContainerDied","Data":"7ffb158edfeb88ff8065594b67ffa93f00336ed27c04dbb5de280a5abddcb719"}
Nov 01 00:54:02.037811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.037626    4493 scope.go:122] "RemoveContainer" containerID="54e3c16ed87ba47027bd17b42e32796751e40d873149f67c532867b39e45a154"
Nov 01 00:54:02.207736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.207701    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:54:02.257165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.257114    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-14957a74-67b5b5b4c9-ntwb4"]
Nov 01 00:54:02.504444 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.504417    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4f1b5c-55ff-400d-9012-97464be1eca2" path="/var/lib/kubelet/pods/2a4f1b5c-55ff-400d-9012-97464be1eca2/volumes"
Nov 01 00:54:02.504896 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:02.504881    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" path="/var/lib/kubelet/pods/e623ec1e-a1df-40f2-9ed3-2855aafbd948/volumes"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.906447    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.907653    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2a4f1b5c-55ff-400d-9012-97464be1eca2" containerName="payload-pre-processing"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.907672    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4f1b5c-55ff-400d-9012-97464be1eca2" containerName="payload-pre-processing"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.907716    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" containerName="payload-processing"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.907774    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" containerName="payload-processing"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.907989    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e623ec1e-a1df-40f2-9ed3-2855aafbd948" containerName="payload-processing"
Nov 01 00:54:03.910158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.908008    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4f1b5c-55ff-400d-9012-97464be1eca2" containerName="payload-pre-processing"
Nov 01 00:54:03.928488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.928463    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:03.996156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:03.989678    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-13f6ab63\""
Nov 01 00:54:04.094046 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.094016    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-13f6ab63-dockercfg-5xq78\""
Nov 01 00:54:04.160274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.103069    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:54:04.192220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.178218    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcqf\" (UniqueName: \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.197354 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.197319    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.308266 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.308127    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:54:04.490234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.462250    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.490234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.462394    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcqf\" (UniqueName: \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.490234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.464171    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.543554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.536995    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcqf\" (UniqueName: \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\") pod \"payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.547615 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.547579    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:54:04.564612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.550296    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.670966 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.670928    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6kwp\" (UniqueName: \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.671485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.671460    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.773760 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.773679    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.773760 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.773746    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6kwp\" (UniqueName: \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.774404 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.774384    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.809776 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.809741    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:04.835106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.835074    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6kwp\" (UniqueName: \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\") pod \"payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") " pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:04.892695 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:04.891970    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:05.194952 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.194927    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:54:05.195794 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:54:05.195772    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faeacc2_671b_49ff_a3a6_e822df292c98.slice/crio-978a27edd9de85ef22844b39d363b36b2fddfd93381970bfb700edc20b46fdaa WatchSource:0}: Error finding container 978a27edd9de85ef22844b39d363b36b2fddfd93381970bfb700edc20b46fdaa: Status 404 returned error can't find the container with id 978a27edd9de85ef22844b39d363b36b2fddfd93381970bfb700edc20b46fdaa
Nov 01 00:54:05.301407 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.301376    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:54:05.301902 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:54:05.301875    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda656de83_5880_4f1b_8d35_5bb073e5a82c.slice/crio-a46dd78ba1f7e6bf7f7f155255fc2e57da0d64597fc9ba7bc7cc758cdfda58c6 WatchSource:0}: Error finding container a46dd78ba1f7e6bf7f7f155255fc2e57da0d64597fc9ba7bc7cc758cdfda58c6: Status 404 returned error can't find the container with id a46dd78ba1f7e6bf7f7f155255fc2e57da0d64597fc9ba7bc7cc758cdfda58c6
Nov 01 00:54:05.475226 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.475183    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" event={"ID":"a656de83-5880-4f1b-8d35-5bb073e5a82c","Type":"ContainerStarted","Data":"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"}
Nov 01 00:54:05.475410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.475250    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" event={"ID":"a656de83-5880-4f1b-8d35-5bb073e5a82c","Type":"ContainerStarted","Data":"a46dd78ba1f7e6bf7f7f155255fc2e57da0d64597fc9ba7bc7cc758cdfda58c6"}
Nov 01 00:54:05.475410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.475292    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:05.476904 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.476878    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" event={"ID":"3faeacc2-671b-49ff-a3a6-e822df292c98","Type":"ContainerStarted","Data":"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"}
Nov 01 00:54:05.477020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.476926    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" event={"ID":"3faeacc2-671b-49ff-a3a6-e822df292c98","Type":"ContainerStarted","Data":"978a27edd9de85ef22844b39d363b36b2fddfd93381970bfb700edc20b46fdaa"}
Nov 01 00:54:05.477020 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.476947    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:05.594502 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.594454    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" podStartSLOduration=2.594438814 podStartE2EDuration="2.594438814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:05.591183457 +0000 UTC m=+1835.268770402" watchObservedRunningTime="2026-11-01 00:54:05.594438814 +0000 UTC m=+1835.272025760"
Nov 01 00:54:05.722525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:05.722456    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" podStartSLOduration=2.722437478 podStartE2EDuration="2.722437478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:05.722008324 +0000 UTC m=+1835.399595270" watchObservedRunningTime="2026-11-01 00:54:05.722437478 +0000 UTC m=+1835.400024426"
Nov 01 00:54:09.028545 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.028513    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:54:09.044567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.044539    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.147544 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.147510    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:54:09.152567 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.152536    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5f10727e\""
Nov 01 00:54:09.152991 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.152974    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5f10727e-dockercfg-8bktm\""
Nov 01 00:54:09.323209 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.268026    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2js\" (UniqueName: \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.356793 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.323537    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.456153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.449445    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2js\" (UniqueName: \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.456153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.449565    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.487156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.480222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.524838 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.521750    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2js\" (UniqueName: \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\") pod \"payload-processing-e2e-derive-5f10727e-6897976994-97hxv\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") " pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:09.765836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:09.765805    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:10.150571 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:10.150542    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:54:10.150994 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:54:10.150824    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae594c08_ef7f_4f17_8d81_9cce9a35388a.slice/crio-986e443010a3858de0d374bd6fdbcba2b066f090fb9b6890599cc1b8e0fb6116 WatchSource:0}: Error finding container 986e443010a3858de0d374bd6fdbcba2b066f090fb9b6890599cc1b8e0fb6116: Status 404 returned error can't find the container with id 986e443010a3858de0d374bd6fdbcba2b066f090fb9b6890599cc1b8e0fb6116
Nov 01 00:54:10.541682 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:10.541649    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" event={"ID":"ae594c08-ef7f-4f17-8d81-9cce9a35388a","Type":"ContainerStarted","Data":"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"}
Nov 01 00:54:10.541876 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:10.541698    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" event={"ID":"ae594c08-ef7f-4f17-8d81-9cce9a35388a","Type":"ContainerStarted","Data":"986e443010a3858de0d374bd6fdbcba2b066f090fb9b6890599cc1b8e0fb6116"}
Nov 01 00:54:10.541876 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:10.541761    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:10.662353 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:10.662304    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" podStartSLOduration=2.662290891 podStartE2EDuration="2.662290891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:10.658430526 +0000 UTC m=+1840.336017471" watchObservedRunningTime="2026-11-01 00:54:10.662290891 +0000 UTC m=+1840.339877842"
Nov 01 00:54:14.990036 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:54:14.989954    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:54:16.490258 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:16.490229    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:54:16.490731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:16.490316    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:54:21.551081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:21.551050    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:54:42.940575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:42.940540    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:54:42.946710 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:42.946686    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.004578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.004541    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:54:43.049474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.049439    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-13f6ab63-serving-cert\""
Nov 01 00:54:43.062280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.062246    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.062768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.062745    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.062905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.062786    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.062905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.062813    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb98q\" (UniqueName: \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.164787 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.164743    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.164787 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.164792    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.165008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.164822    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb98q\" (UniqueName: \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.165008 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:54:43.164914    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-serving-cert: secret "maas-api-e2e-ait-13f6ab63-serving-cert" not found
Nov 01 00:54:43.165008 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:54:43.164998    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls podName:4040f81b-459c-499a-a18e-ebdfa13d7e26 nodeName:}" failed. No retries permitted until 2026-11-01 00:54:43.664977227 +0000 UTC m=+1873.342564169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls") pod "maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" (UID: "4040f81b-459c-499a-a18e-ebdfa13d7e26") : secret "maas-api-e2e-ait-13f6ab63-serving-cert" not found
Nov 01 00:54:43.165113 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.165020    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.165616 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.165592    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.167581 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.167563    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.248678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.242627    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb98q\" (UniqueName: \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.656700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.656618    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:54:43.678097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.678067    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.680746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.680718    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") pod \"maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:43.861678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:43.861641    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:44.297702 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:44.297673    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:54:44.298125 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:54:44.297992    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4040f81b_459c_499a_a18e_ebdfa13d7e26.slice/crio-f4e2d5c5e2c7101b0337b2b6fe814b12625e63c1fc5a2f994d03f85c12b582c7 WatchSource:0}: Error finding container f4e2d5c5e2c7101b0337b2b6fe814b12625e63c1fc5a2f994d03f85c12b582c7: Status 404 returned error can't find the container with id f4e2d5c5e2c7101b0337b2b6fe814b12625e63c1fc5a2f994d03f85c12b582c7
Nov 01 00:54:44.785489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:44.785453    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" event={"ID":"4040f81b-459c-499a-a18e-ebdfa13d7e26","Type":"ContainerStarted","Data":"f4e2d5c5e2c7101b0337b2b6fe814b12625e63c1fc5a2f994d03f85c12b582c7"}
Nov 01 00:54:44.991487 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:54:44.991440    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:54:45.793385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:45.793344    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" event={"ID":"4040f81b-459c-499a-a18e-ebdfa13d7e26","Type":"ContainerStarted","Data":"35a3dcedf57854c260c2a4e22dd3c5b2f09a7c71530820317e1385077513620b"}
Nov 01 00:54:45.793791 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:45.793447    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:45.912419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:45.912363    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" podStartSLOduration=3.032165418 podStartE2EDuration="3.912348767s" totalImagesPullingTime="880.183349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:45.910623325 +0000 UTC m=+1875.588210270" watchObservedRunningTime="2026-11-01 00:54:45.912348767 +0000 UTC m=+1875.589935711"
Nov 01 00:54:47.924855 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:47.924823    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:54:47.929292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:47.929273    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:47.983165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:47.983096    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5f10727e-serving-cert\""
Nov 01 00:54:47.988563 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:47.988533    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:54:48.053433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.053399    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.053605 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.053440    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcr5k\" (UniqueName: \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.053672 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.053658    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.053757 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.053742    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.155013 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.154978    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.155242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.155110    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.155242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.155179    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcr5k\" (UniqueName: \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.155473 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.155449    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.156276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.156252    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.157827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.157804    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.157938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.157926    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.216315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.216250    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcr5k\" (UniqueName: \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\") pod \"maas-api-e2e-derive-5f10727e-79779df49f-jlxdp\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.244503 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.244475    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:48.665621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.665592    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:54:48.666163 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:54:48.666104    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645a6b9f_8446_454c_aa57_2fb62d6896d4.slice/crio-0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f WatchSource:0}: Error finding container 0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f: Status 404 returned error can't find the container with id 0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f
Nov 01 00:54:48.818773 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:48.818737    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" event={"ID":"645a6b9f-8446-454c-aa57-2fb62d6896d4","Type":"ContainerStarted","Data":"0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f"}
Nov 01 00:54:49.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:49.829112    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" event={"ID":"645a6b9f-8446-454c-aa57-2fb62d6896d4","Type":"ContainerStarted","Data":"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"}
Nov 01 00:54:49.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:49.829208    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:54:49.948885 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:49.948838    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" podStartSLOduration=2.1013618960000002 podStartE2EDuration="2.948823625s" totalImagesPullingTime="847.461729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:49.946028027 +0000 UTC m=+1879.623614984" watchObservedRunningTime="2026-11-01 00:54:49.948823625 +0000 UTC m=+1879.626410569"
Nov 01 00:54:51.807938 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:51.807913    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:54:55.845514 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:54:55.845484    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:55:04.404180 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.404118    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:55:04.410393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.410368    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.466836 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.466787    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:55:04.539110 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.539028    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a31bcdff-dockercfg-hfxbc\""
Nov 01 00:55:04.580243 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.546303    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a31bcdff\""
Nov 01 00:55:04.647255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.610240    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.686219 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.647818    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttnh\" (UniqueName: \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.879803 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.879584    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.881325 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.881299    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttnh\" (UniqueName: \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.881957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.880990    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.932380 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.932338    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:55:04.955086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.954333    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:55:04.955086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.954439    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:04.966621 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.963808    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttnh\" (UniqueName: \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\") pod \"payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:04.983610 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.983579    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwbx\" (UniqueName: \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:04.983736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:04.983626    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.026625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.026593    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:05.088112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.086714    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwbx\" (UniqueName: \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.088112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.086770    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.088112 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.087989    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.150407 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.150374    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwbx\" (UniqueName: \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\") pod \"payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") " pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.288492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.288459    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:05.429989 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.429959    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:55:05.680042 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:55:05.680015    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb191a478_5321_4b31_9811_0c6c8b142483.slice/crio-21f8c39bc06887d8e3e879783c9551629924b0cda0d3aae2f798e1265f27ba23 WatchSource:0}: Error finding container 21f8c39bc06887d8e3e879783c9551629924b0cda0d3aae2f798e1265f27ba23: Status 404 returned error can't find the container with id 21f8c39bc06887d8e3e879783c9551629924b0cda0d3aae2f798e1265f27ba23
Nov 01 00:55:05.680176 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:05.680122    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:55:06.135050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.131434    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" event={"ID":"b191a478-5321-4b31-9811-0c6c8b142483","Type":"ContainerStarted","Data":"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"}
Nov 01 00:55:06.135050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.131501    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" event={"ID":"b191a478-5321-4b31-9811-0c6c8b142483","Type":"ContainerStarted","Data":"21f8c39bc06887d8e3e879783c9551629924b0cda0d3aae2f798e1265f27ba23"}
Nov 01 00:55:06.135050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.132673    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:06.147166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.143609    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" event={"ID":"e3e61b05-6a17-4cf0-be39-65d688c3f926","Type":"ContainerStarted","Data":"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"}
Nov 01 00:55:06.147166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.143686    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" event={"ID":"e3e61b05-6a17-4cf0-be39-65d688c3f926","Type":"ContainerStarted","Data":"7b06f6b07ba64c5d37353ea3676231a5f52c399e6f904bcfcf78064975de3709"}
Nov 01 00:55:06.147166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.144690    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:06.279156 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.275169    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" podStartSLOduration=2.2751495569999998 podStartE2EDuration="2.275149557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:06.271062657 +0000 UTC m=+1895.948649603" watchObservedRunningTime="2026-11-01 00:55:06.275149557 +0000 UTC m=+1895.952736529"
Nov 01 00:55:06.396255 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:06.394860    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" podStartSLOduration=2.394839101 podStartE2EDuration="2.394839101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:06.389863614 +0000 UTC m=+1896.067450561" watchObservedRunningTime="2026-11-01 00:55:06.394839101 +0000 UTC m=+1896.072426047"
Nov 01 00:55:14.991952 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:55:14.991921    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:55:16.062307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:16.062262    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 00:55:16.062820 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:16.062341    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:55:17.335755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.335663    4493 generic.go:356] "Generic (PLEG): container finished" podID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerID="bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870" exitCode=0
Nov 01 00:55:17.335755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.335736    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerDied","Data":"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"}
Nov 01 00:55:17.336312 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.336234    4493 scope.go:122] "RemoveContainer" containerID="bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"
Nov 01 00:55:17.337898 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.337869    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:55:17.337957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.337940    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:55:17.337992 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:17.337972    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:55:18.251884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:18.251852    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:55:18.252063 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:18.251922    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:55:18.347831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:18.347797    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerStarted","Data":"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"}
Nov 01 00:55:19.140436 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:19.140393    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:55:19.140653 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:19.140632    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:55:19.140733 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:19.140687    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:55:20.140292 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:20.140249    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:55:20.140736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:20.140327    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:55:21.139158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:21.136591    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:55:21.139158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:21.136667    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:55:22.124166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:22.119550    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:55:22.124166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:22.119620    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:55:23.131159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:23.119523    4493 patch_prober.go:29] interesting pod/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 00:55:23.131159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:23.119593    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 00:55:24.121597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:24.121568    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:55:24.122057 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:24.122037    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:55:24.122619 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:24.122603    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:55:38.755435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:38.755400    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:55:40.183536 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:40.183507    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:55:40.183985 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:40.183696    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" containerID="cri-o://e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43" gracePeriod=30
Nov 01 00:55:43.695535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.695222    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 00:55:43.732157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.731476    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:43.782738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.782276    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 00:55:43.789192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.789117    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-99eb0f-dockercfg-9gsv5\""
Nov 01 00:55:43.961212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.885872    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-99eb0f\""
Nov 01 00:55:43.970151 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.911713    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9xf\" (UniqueName: \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:43.970582 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.970549    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:43.991671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:43.946039    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 00:55:44.014865 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.014831    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 00:55:44.015249 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.015234    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.109449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.100261    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrs5\" (UniqueName: \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.109449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.100422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xf\" (UniqueName: \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:44.109449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.100462    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.109449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.100543    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:44.109449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.101711    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:44.258161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.254905    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrs5\" (UniqueName: \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.258161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.255097    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.258161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.256687    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.287243 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.282249    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9xf\" (UniqueName: \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\") pod \"payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:44.318392 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.318364    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrs5\" (UniqueName: \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\") pod \"payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.530962 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.530881    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:44.601211 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.601177    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:44.685100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.685073    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:55:44.685276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.685119    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:55:44.913365 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:55:44.913332    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fe46e4_f198_4140_a17b_e13cb853ad2b.slice/crio-877e3fa9860ae1f38b9d2b70d2b6e959a85c14d7c3ae45b0b6d64c0ef30c307b WatchSource:0}: Error finding container 877e3fa9860ae1f38b9d2b70d2b6e959a85c14d7c3ae45b0b6d64c0ef30c307b: Status 404 returned error can't find the container with id 877e3fa9860ae1f38b9d2b70d2b6e959a85c14d7c3ae45b0b6d64c0ef30c307b
Nov 01 00:55:44.914661 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:44.914634    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 00:55:44.994780 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:55:44.994748    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:55:45.013639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.013612    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 00:55:45.014271 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:55:45.014249    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9da717_8e34_4631_94c8_a7c763ca8dc4.slice/crio-20ed81fc35e9161b759230d4a9cbde2972c4523d8620aab32194f33322181f3b WatchSource:0}: Error finding container 20ed81fc35e9161b759230d4a9cbde2972c4523d8620aab32194f33322181f3b: Status 404 returned error can't find the container with id 20ed81fc35e9161b759230d4a9cbde2972c4523d8620aab32194f33322181f3b
Nov 01 00:55:45.455510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.455446    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:55:45.575656 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575625    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.575880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575780    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.575880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575826    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.575880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575855    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575892    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575914    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:55:45.576061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.575996    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576061 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576032    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576086    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576168    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:55:45.576328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576203    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\") pod \"14011b58-4d69-4210-985a-485921ffe69e\" (UID: \"14011b58-4d69-4210-985a-485921ffe69e\") "
Nov 01 00:55:45.576475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576321    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:55:45.576475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576432    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:55:45.576743 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.576606    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:55:45.578062 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.578035    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.578224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.578064    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.578224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.578080    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14011b58-4d69-4210-985a-485921ffe69e-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.578224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.578094    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.578224 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.578107    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.579844 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.579778    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:55:45.580218 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.580191    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "kube-api-access-t9f9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:55:45.580419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.580385    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:55:45.580614 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.580592    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy" pod "14011b58-4d69-4210-985a-485921ffe69e" (UID: "14011b58-4d69-4210-985a-485921ffe69e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:55:45.677262 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.677234    4493 generic.go:356] "Generic (PLEG): container finished" podID="14011b58-4d69-4210-985a-485921ffe69e" containerID="e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43" exitCode=0
Nov 01 00:55:45.677474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.677304    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"
Nov 01 00:55:45.677474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.677323    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" event={"ID":"14011b58-4d69-4210-985a-485921ffe69e","Type":"ContainerDied","Data":"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"}
Nov 01 00:55:45.677474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.677388    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg" event={"ID":"14011b58-4d69-4210-985a-485921ffe69e","Type":"ContainerDied","Data":"0f886b115a53747737e784b8af3d9f0588543b46856e9ee42bdb52e517b2a131"}
Nov 01 00:55:45.677474 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.677419    4493 scope.go:122] "RemoveContainer" containerID="e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"
Nov 01 00:55:45.679398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679372    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679401    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14011b58-4d69-4210-985a-485921ffe69e-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679409    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" event={"ID":"01fe46e4-f198-4140-a17b-e13cb853ad2b","Type":"ContainerStarted","Data":"7c020b4dd07cff7f62b8c52b50c22dddfb8144e3665df2839253459fa8de7cf9"}
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679416    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14011b58-4d69-4210-985a-485921ffe69e-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679451    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" event={"ID":"01fe46e4-f198-4140-a17b-e13cb853ad2b","Type":"ContainerStarted","Data":"877e3fa9860ae1f38b9d2b70d2b6e959a85c14d7c3ae45b0b6d64c0ef30c307b"}
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679472    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9f9z\" (UniqueName: \"kubernetes.io/projected/14011b58-4d69-4210-985a-485921ffe69e-kube-api-access-t9f9z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:55:45.679527 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.679477    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:45.681422 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.681398    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" event={"ID":"ce9da717-8e34-4631-94c8-a7c763ca8dc4","Type":"ContainerStarted","Data":"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"}
Nov 01 00:55:45.681546 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.681444    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" event={"ID":"ce9da717-8e34-4631-94c8-a7c763ca8dc4","Type":"ContainerStarted","Data":"20ed81fc35e9161b759230d4a9cbde2972c4523d8620aab32194f33322181f3b"}
Nov 01 00:55:45.681546 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.681473    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:55:45.711317 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.710823    4493 scope.go:122] "RemoveContainer" containerID="e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"
Nov 01 00:55:45.713399 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:55:45.712220    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43\": container with ID starting with e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43 not found: ID does not exist" containerID="e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"
Nov 01 00:55:45.713399 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.712263    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43"} err="failed to get container status \"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43\": rpc error: code = NotFound desc = could not find container \"e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43\": container with ID starting with e338419ee68ec680d4b2e94e982d31785cd303c47ec61dbc9520c577284ecf43 not found: ID does not exist"
Nov 01 00:55:45.849295 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.849245    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" podStartSLOduration=2.849231131 podStartE2EDuration="2.849231131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:45.84579096 +0000 UTC m=+1935.523377905" watchObservedRunningTime="2026-11-01 00:55:45.849231131 +0000 UTC m=+1935.526818101"
Nov 01 00:55:45.989210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:45.983346    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" podStartSLOduration=2.983314912 podStartE2EDuration="2.983314912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:45.972722604 +0000 UTC m=+1935.650309554" watchObservedRunningTime="2026-11-01 00:55:45.983314912 +0000 UTC m=+1935.660901857"
Nov 01 00:55:46.173831 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:46.173761    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:55:46.214782 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:46.214744    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-z7hkg"]
Nov 01 00:55:46.504720 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:46.504689    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14011b58-4d69-4210-985a-485921ffe69e" path="/var/lib/kubelet/pods/14011b58-4d69-4210-985a-485921ffe69e/volumes"
Nov 01 00:55:50.054475 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:50.054442    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"]
Nov 01 00:55:50.104305 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:50.104271    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd01240-ww5cx"]
Nov 01 00:55:50.506351 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:50.506318    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d0c42a-95f8-49e0-b74e-5658509a1c82" path="/var/lib/kubelet/pods/19d0c42a-95f8-49e0-b74e-5658509a1c82/volumes"
Nov 01 00:55:53.412793 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.412762    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:55:53.413553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.413537    4493 cpu_manager.go:414] "containerMap: removing container" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy"
Nov 01 00:55:53.413612 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.413554    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy"
Nov 01 00:55:53.413669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.413658    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="14011b58-4d69-4210-985a-485921ffe69e" containerName="istio-proxy"
Nov 01 00:55:53.416093 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.416072    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.468568 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.468532    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a31bcdff-serving-cert\""
Nov 01 00:55:53.523172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.523108    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:55:53.547738 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.547705    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.548009 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.547991    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzgt\" (UniqueName: \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.548094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.548028    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.548094 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.548051    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.651539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.649677    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzgt\" (UniqueName: \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.651539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.649741    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.651539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.649773    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.651539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.649902    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.651539 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.650810    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.678205 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:55:53.666822    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-serving-cert: secret "maas-api-e2e-mt-a31bcdff-serving-cert" not found
Nov 01 00:55:53.678205 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:55:53.666923    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls podName:51b6791a-8750-4f5b-aae3-68a92a372b00 nodeName:}" failed. No retries permitted until 2026-11-01 00:55:54.166900381 +0000 UTC m=+1943.844487321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls") pod "maas-api-e2e-mt-a31bcdff-5496c658b-587gf" (UID: "51b6791a-8750-4f5b-aae3-68a92a372b00") : secret "maas-api-e2e-mt-a31bcdff-serving-cert" not found
Nov 01 00:55:53.709402 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.696076    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:53.764164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:53.755863    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzgt\" (UniqueName: \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:54.267817 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:54.267784    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:54.270271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:54.270245    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") pod \"maas-api-e2e-mt-a31bcdff-5496c658b-587gf\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:54.330048 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:54.330017    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:54.727152 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:54.723676    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:55:54.775967 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:54.775929    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" event={"ID":"51b6791a-8750-4f5b-aae3-68a92a372b00","Type":"ContainerStarted","Data":"cfe2c33ce38f3688d80d87d298eda2b38148c95caf02bcf2539184f61fcef722"}
Nov 01 00:55:55.786905 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:55.786867    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" event={"ID":"51b6791a-8750-4f5b-aae3-68a92a372b00","Type":"ContainerStarted","Data":"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"}
Nov 01 00:55:55.787394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:55.786969    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:55:55.919982 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:55.919871    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" podStartSLOduration=1.984747169 podStartE2EDuration="2.919852465s" totalImagesPullingTime="935.105296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:55.916183643 +0000 UTC m=+1945.593770586" watchObservedRunningTime="2026-11-01 00:55:55.919852465 +0000 UTC m=+1945.597439414"
Nov 01 00:55:56.203664 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:56.203575    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:55:56.326099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:56.326068    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:55:56.992111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:56.992084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 00:55:56.992737 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:55:56.992717    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 00:56:01.204850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.204803    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" podUID="a656de83-5880-4f1b-8d35-5bb073e5a82c" containerName="payload-processing" containerID="cri-o://2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9" gracePeriod=25
Nov 01 00:56:01.327015 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.326980    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" podUID="3faeacc2-671b-49ff-a3a6-e822df292c98" containerName="payload-pre-processing" containerID="cri-o://606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c" gracePeriod=25
Nov 01 00:56:01.535595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.535563    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:56:01.556176 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.556120    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\" (UniqueName: \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\") pod \"a656de83-5880-4f1b-8d35-5bb073e5a82c\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") "
Nov 01 00:56:01.556405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.556352    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\") pod \"a656de83-5880-4f1b-8d35-5bb073e5a82c\" (UID: \"a656de83-5880-4f1b-8d35-5bb073e5a82c\") "
Nov 01 00:56:01.556882 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.556838    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume" pod "a656de83-5880-4f1b-8d35-5bb073e5a82c" (UID: "a656de83-5880-4f1b-8d35-5bb073e5a82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:01.557442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.557420    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a656de83-5880-4f1b-8d35-5bb073e5a82c-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:01.560796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.560774    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp" pod "a656de83-5880-4f1b-8d35-5bb073e5a82c" (UID: "a656de83-5880-4f1b-8d35-5bb073e5a82c"). InnerVolumeSpecName "kube-api-access-k6kwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:01.606659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.606631    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:56:01.658651 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.658620    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\") pod \"3faeacc2-671b-49ff-a3a6-e822df292c98\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") "
Nov 01 00:56:01.658960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.658943    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\" (UniqueName: \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\") pod \"3faeacc2-671b-49ff-a3a6-e822df292c98\" (UID: \"3faeacc2-671b-49ff-a3a6-e822df292c98\") "
Nov 01 00:56:01.659058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.659018    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume" pod "3faeacc2-671b-49ff-a3a6-e822df292c98" (UID: "3faeacc2-671b-49ff-a3a6-e822df292c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:01.659397 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.659383    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3faeacc2-671b-49ff-a3a6-e822df292c98-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:01.659442 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.659400    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6kwp\" (UniqueName: \"kubernetes.io/projected/a656de83-5880-4f1b-8d35-5bb073e5a82c-kube-api-access-k6kwp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:01.661323 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.661295    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf" pod "3faeacc2-671b-49ff-a3a6-e822df292c98" (UID: "3faeacc2-671b-49ff-a3a6-e822df292c98"). InnerVolumeSpecName "kube-api-access-wtcqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:01.761365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.761289    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtcqf\" (UniqueName: \"kubernetes.io/projected/3faeacc2-671b-49ff-a3a6-e822df292c98-kube-api-access-wtcqf\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:01.999645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:01.999613    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:56:02.072494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.072411    4493 generic.go:356] "Generic (PLEG): container finished" podID="a656de83-5880-4f1b-8d35-5bb073e5a82c" containerID="2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9" exitCode=0
Nov 01 00:56:02.072642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.072507    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" event={"ID":"a656de83-5880-4f1b-8d35-5bb073e5a82c","Type":"ContainerDied","Data":"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"}
Nov 01 00:56:02.072642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.072522    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"
Nov 01 00:56:02.072642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.072564    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg" event={"ID":"a656de83-5880-4f1b-8d35-5bb073e5a82c","Type":"ContainerDied","Data":"a46dd78ba1f7e6bf7f7f155255fc2e57da0d64597fc9ba7bc7cc758cdfda58c6"}
Nov 01 00:56:02.072642 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.072595    4493 scope.go:122] "RemoveContainer" containerID="2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"
Nov 01 00:56:02.074019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.073997    4493 generic.go:356] "Generic (PLEG): container finished" podID="3faeacc2-671b-49ff-a3a6-e822df292c98" containerID="606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c" exitCode=0
Nov 01 00:56:02.074125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.074042    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"
Nov 01 00:56:02.074125 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.074093    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" event={"ID":"3faeacc2-671b-49ff-a3a6-e822df292c98","Type":"ContainerDied","Data":"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"}
Nov 01 00:56:02.074296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.074249    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f" event={"ID":"3faeacc2-671b-49ff-a3a6-e822df292c98","Type":"ContainerDied","Data":"978a27edd9de85ef22844b39d363b36b2fddfd93381970bfb700edc20b46fdaa"}
Nov 01 00:56:02.084962 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.084944    4493 scope.go:122] "RemoveContainer" containerID="2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"
Nov 01 00:56:02.085407 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:02.085382    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9\": container with ID starting with 2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9 not found: ID does not exist" containerID="2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"
Nov 01 00:56:02.085490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.085420    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9"} err="failed to get container status \"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9\": rpc error: code = NotFound desc = could not find container \"2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9\": container with ID starting with 2b911ecf74b2094c26bb336339f493c78ee108600294a540d8f3740ac1ddd8d9 not found: ID does not exist"
Nov 01 00:56:02.085490 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.085446    4493 scope.go:122] "RemoveContainer" containerID="606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"
Nov 01 00:56:02.095419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.095400    4493 scope.go:122] "RemoveContainer" containerID="606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"
Nov 01 00:56:02.095694 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:02.095675    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c\": container with ID starting with 606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c not found: ID does not exist" containerID="606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"
Nov 01 00:56:02.095736 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.095704    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c"} err="failed to get container status \"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c\": rpc error: code = NotFound desc = could not find container \"606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c\": container with ID starting with 606288058f615347a70c144e695c26c6deb28ae1ece2956f9b3d8a5a3eb2cb6c not found: ID does not exist"
Nov 01 00:56:02.235948 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.235903    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:56:02.265964 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.262808    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13f6ab63-bc59dd795-9pv9f"]
Nov 01 00:56:02.357310 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.357225    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:56:02.363536 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.363504    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13f6ab63-55b79cd674-8cwlg"]
Nov 01 00:56:02.504872 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.504837    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faeacc2-671b-49ff-a3a6-e822df292c98" path="/var/lib/kubelet/pods/3faeacc2-671b-49ff-a3a6-e822df292c98/volumes"
Nov 01 00:56:02.505360 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:02.505346    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a656de83-5880-4f1b-8d35-5bb073e5a82c" path="/var/lib/kubelet/pods/a656de83-5880-4f1b-8d35-5bb073e5a82c/volumes"
Nov 01 00:56:03.520180 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:03.515932    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:56:08.532750 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:08.532696    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" podUID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" containerName="payload-processing" containerID="cri-o://2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9" gracePeriod=25
Nov 01 00:56:09.168604 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.168577    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:56:09.227431 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.227398    4493 generic.go:356] "Generic (PLEG): container finished" podID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" containerID="2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9" exitCode=0
Nov 01 00:56:09.227597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.227447    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"
Nov 01 00:56:09.227597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.227471    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" event={"ID":"ae594c08-ef7f-4f17-8d81-9cce9a35388a","Type":"ContainerDied","Data":"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"}
Nov 01 00:56:09.227597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.227531    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv" event={"ID":"ae594c08-ef7f-4f17-8d81-9cce9a35388a","Type":"ContainerDied","Data":"986e443010a3858de0d374bd6fdbcba2b066f090fb9b6890599cc1b8e0fb6116"}
Nov 01 00:56:09.227597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.227556    4493 scope.go:122] "RemoveContainer" containerID="2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"
Nov 01 00:56:09.237695 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.237671    4493 scope.go:122] "RemoveContainer" containerID="2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"
Nov 01 00:56:09.237982 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:09.237965    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9\": container with ID starting with 2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9 not found: ID does not exist" containerID="2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"
Nov 01 00:56:09.238038 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.237991    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9"} err="failed to get container status \"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9\": rpc error: code = NotFound desc = could not find container \"2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9\": container with ID starting with 2b22d32b313a6d4e6aa899229c27b7a54b308adfdc7d0cf41137c401bb0ecfe9 not found: ID does not exist"
Nov 01 00:56:09.250488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.250455    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\") pod \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") "
Nov 01 00:56:09.250837 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.250817    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\" (UniqueName: \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\") pod \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\" (UID: \"ae594c08-ef7f-4f17-8d81-9cce9a35388a\") "
Nov 01 00:56:09.250945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.250818    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume" pod "ae594c08-ef7f-4f17-8d81-9cce9a35388a" (UID: "ae594c08-ef7f-4f17-8d81-9cce9a35388a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:09.251428 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.251412    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae594c08-ef7f-4f17-8d81-9cce9a35388a-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:09.252972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.252951    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js" pod "ae594c08-ef7f-4f17-8d81-9cce9a35388a" (UID: "ae594c08-ef7f-4f17-8d81-9cce9a35388a"). InnerVolumeSpecName "kube-api-access-xm2js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:09.352711 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.352671    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm2js\" (UniqueName: \"kubernetes.io/projected/ae594c08-ef7f-4f17-8d81-9cce9a35388a-kube-api-access-xm2js\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:09.661234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.661197    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:56:09.665419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:09.665387    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5f10727e-6897976994-97hxv"]
Nov 01 00:56:10.504589 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:10.504557    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" path="/var/lib/kubelet/pods/ae594c08-ef7f-4f17-8d81-9cce9a35388a/volumes"
Nov 01 00:56:14.990250 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:14.990210    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:56:17.235394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.235355    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 00:56:17.237931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.237886    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" containerName="payload-processing"
Nov 01 00:56:17.238158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238119    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" containerName="payload-processing"
Nov 01 00:56:17.238307 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238293    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3faeacc2-671b-49ff-a3a6-e822df292c98" containerName="payload-pre-processing"
Nov 01 00:56:17.238394 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238383    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faeacc2-671b-49ff-a3a6-e822df292c98" containerName="payload-pre-processing"
Nov 01 00:56:17.238481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238471    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a656de83-5880-4f1b-8d35-5bb073e5a82c" containerName="payload-processing"
Nov 01 00:56:17.238557 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238546    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a656de83-5880-4f1b-8d35-5bb073e5a82c" containerName="payload-processing"
Nov 01 00:56:17.238886 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.238868    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae594c08-ef7f-4f17-8d81-9cce9a35388a" containerName="payload-processing"
Nov 01 00:56:17.239026 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.239009    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faeacc2-671b-49ff-a3a6-e822df292c98" containerName="payload-pre-processing"
Nov 01 00:56:17.239155 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.239124    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a656de83-5880-4f1b-8d35-5bb073e5a82c" containerName="payload-processing"
Nov 01 00:56:17.253582 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.253538    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.336279 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.336243    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 00:56:17.352467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.352432    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b112c158\""
Nov 01 00:56:17.607597 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.607483    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 00:56:17.619357 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.619293    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.659586 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.659544    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6x8q\" (UniqueName: \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.660550 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.656427    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.717181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.714200    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 00:56:17.736453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.719291    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b112c158-dockercfg-f5dkx\""
Nov 01 00:56:17.820053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.813290    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.820053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.814728    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.820053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.814850    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.820053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.814979    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6x8q\" (UniqueName: \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.820053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.815351    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshrk\" (UniqueName: \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.877718 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.877651    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6x8q\" (UniqueName: \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.916846 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.916796    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.917030 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.917012    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshrk\" (UniqueName: \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.917578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.917557    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:17.925332 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.925312    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:17.978809 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:17.978735    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshrk\" (UniqueName: \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:18.144283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:18.144194    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:18.305795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:18.305765    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 00:56:18.306740 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:18.306710    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722513e5_97ca_41c3_95df_6cddd56cb1d7.slice/crio-ed853d4ab57a57f08c6105d03ea02c1fafb017a971ac71d52f60d1d345c04b3a WatchSource:0}: Error finding container ed853d4ab57a57f08c6105d03ea02c1fafb017a971ac71d52f60d1d345c04b3a: Status 404 returned error can't find the container with id ed853d4ab57a57f08c6105d03ea02c1fafb017a971ac71d52f60d1d345c04b3a
Nov 01 00:56:18.324903 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:18.324875    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" event={"ID":"722513e5-97ca-41c3-95df-6cddd56cb1d7","Type":"ContainerStarted","Data":"ed853d4ab57a57f08c6105d03ea02c1fafb017a971ac71d52f60d1d345c04b3a"}
Nov 01 00:56:18.525433 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:18.525405    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 00:56:18.525720 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:18.525685    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec90e9f_68ab_49cd_85de_4521d65aff26.slice/crio-dd751958efd64d4c4c511092d9c6ced4497f764e097574074696ca62b3c21058 WatchSource:0}: Error finding container dd751958efd64d4c4c511092d9c6ced4497f764e097574074696ca62b3c21058: Status 404 returned error can't find the container with id dd751958efd64d4c4c511092d9c6ced4497f764e097574074696ca62b3c21058
Nov 01 00:56:19.333761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.333722    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" event={"ID":"7ec90e9f-68ab-49cd-85de-4521d65aff26","Type":"ContainerStarted","Data":"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"}
Nov 01 00:56:19.334275 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.333793    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:19.334275 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.333816    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" event={"ID":"7ec90e9f-68ab-49cd-85de-4521d65aff26","Type":"ContainerStarted","Data":"dd751958efd64d4c4c511092d9c6ced4497f764e097574074696ca62b3c21058"}
Nov 01 00:56:19.335414 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.335389    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" event={"ID":"722513e5-97ca-41c3-95df-6cddd56cb1d7","Type":"ContainerStarted","Data":"37c8b91ef0af284fbeb359b88f345a2c5af4ec751f104229cfc3d0f5ef6a111c"}
Nov 01 00:56:19.335543 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.335466    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:19.497533 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.497487    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" podStartSLOduration=2.497471878 podStartE2EDuration="2.497471878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:19.494309685 +0000 UTC m=+1969.171896629" watchObservedRunningTime="2026-11-01 00:56:19.497471878 +0000 UTC m=+1969.175058814"
Nov 01 00:56:19.632172 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:19.632027    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" podStartSLOduration=2.632003623 podStartE2EDuration="2.632003623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:19.615055169 +0000 UTC m=+1969.292642112" watchObservedRunningTime="2026-11-01 00:56:19.632003623 +0000 UTC m=+1969.309590569"
Nov 01 00:56:20.634809 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:20.634769    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:56:25.543828 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:25.543792    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:56:25.544247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:25.544044    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" containerID="cri-o://10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74" gracePeriod=30
Nov 01 00:56:28.093153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.093097    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:56:28.093628 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.093358    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" podUID="4040f81b-459c-499a-a18e-ebdfa13d7e26" containerName="maas-api" containerID="cri-o://35a3dcedf57854c260c2a4e22dd3c5b2f09a7c71530820317e1385077513620b" gracePeriod=30
Nov 01 00:56:28.653481 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.653343    4493 generic.go:356] "Generic (PLEG): container finished" podID="4040f81b-459c-499a-a18e-ebdfa13d7e26" containerID="35a3dcedf57854c260c2a4e22dd3c5b2f09a7c71530820317e1385077513620b" exitCode=0
Nov 01 00:56:28.663157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.656331    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:56:28.663157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.657897    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\") pod \"4040f81b-459c-499a-a18e-ebdfa13d7e26\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") "
Nov 01 00:56:28.663157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.658001    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\" (UniqueName: \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\") pod \"4040f81b-459c-499a-a18e-ebdfa13d7e26\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") "
Nov 01 00:56:28.674208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.673761    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca" pod "4040f81b-459c-499a-a18e-ebdfa13d7e26" (UID: "4040f81b-459c-499a-a18e-ebdfa13d7e26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:28.688386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.686812    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\") pod \"4040f81b-459c-499a-a18e-ebdfa13d7e26\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") "
Nov 01 00:56:28.688386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.686913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") pod \"4040f81b-459c-499a-a18e-ebdfa13d7e26\" (UID: \"4040f81b-459c-499a-a18e-ebdfa13d7e26\") "
Nov 01 00:56:28.688386 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.687842    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4040f81b-459c-499a-a18e-ebdfa13d7e26-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:28.689604 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.689575    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs" pod "4040f81b-459c-499a-a18e-ebdfa13d7e26" (UID: "4040f81b-459c-499a-a18e-ebdfa13d7e26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:56:28.689790 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.689769    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q" pod "4040f81b-459c-499a-a18e-ebdfa13d7e26" (UID: "4040f81b-459c-499a-a18e-ebdfa13d7e26"). InnerVolumeSpecName "kube-api-access-qb98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:28.702721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.698539    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" event={"ID":"4040f81b-459c-499a-a18e-ebdfa13d7e26","Type":"ContainerDied","Data":"35a3dcedf57854c260c2a4e22dd3c5b2f09a7c71530820317e1385077513620b"}
Nov 01 00:56:28.702721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.698618    4493 scope.go:122] "RemoveContainer" containerID="35a3dcedf57854c260c2a4e22dd3c5b2f09a7c71530820317e1385077513620b"
Nov 01 00:56:28.704525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.704491    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls" pod "4040f81b-459c-499a-a18e-ebdfa13d7e26" (UID: "4040f81b-459c-499a-a18e-ebdfa13d7e26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:56:28.793606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.791666    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:28.793606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.791705    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb98q\" (UniqueName: \"kubernetes.io/projected/4040f81b-459c-499a-a18e-ebdfa13d7e26-kube-api-access-qb98q\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:28.793606 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:28.791722    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4040f81b-459c-499a-a18e-ebdfa13d7e26-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:29.661128 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.661094    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"
Nov 01 00:56:29.661553 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.661125    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf" event={"ID":"4040f81b-459c-499a-a18e-ebdfa13d7e26","Type":"ContainerDied","Data":"f4e2d5c5e2c7101b0337b2b6fe814b12625e63c1fc5a2f994d03f85c12b582c7"}
Nov 01 00:56:29.753328 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.753293    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 00:56:29.754073 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.754053    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4040f81b-459c-499a-a18e-ebdfa13d7e26" containerName="maas-api"
Nov 01 00:56:29.754211 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.754074    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4040f81b-459c-499a-a18e-ebdfa13d7e26" containerName="maas-api"
Nov 01 00:56:29.754276 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.754250    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4040f81b-459c-499a-a18e-ebdfa13d7e26" containerName="maas-api"
Nov 01 00:56:29.757835 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.757816    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.813246 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.813218    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-99eb0f-serving-cert\""
Nov 01 00:56:29.819364 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.819328    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 00:56:29.819691 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.819669    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.819797 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.819784    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.819997 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.819975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.820258 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.820235    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwbq\" (UniqueName: \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.828894 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.828867    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:56:29.835084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.835054    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13f6ab63-6965d6f7f7-v22tf"]
Nov 01 00:56:29.922285 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.922182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.922454 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.922406    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwbq\" (UniqueName: \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.922526 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.922472    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.922584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.922568    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.922820 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:29.922794    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-serving-cert: secret "maas-api-e2e-shared-a-w5-99eb0f-serving-cert" not found
Nov 01 00:56:29.922931 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:29.922870    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls podName:b57b67b9-dda2-46a4-a9b3-f38d49c18f13 nodeName:}" failed. No retries permitted until 2026-11-01 00:56:30.422848214 +0000 UTC m=+1980.100435151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls") pod "maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" (UID: "b57b67b9-dda2-46a4-a9b3-f38d49c18f13") : secret "maas-api-e2e-shared-a-w5-99eb0f-serving-cert" not found
Nov 01 00:56:29.923832 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.923808    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.925754 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.925731    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:29.984859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:29.984824    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwbq\" (UniqueName: \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:30.405574 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.405543    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 00:56:30.405785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.405674    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 00:56:30.420124 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.420090    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:56:30.420124 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.420158    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:56:30.447233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.447200    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:30.449809 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.449788    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:30.505090 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.505061    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4040f81b-459c-499a-a18e-ebdfa13d7e26" path="/var/lib/kubelet/pods/4040f81b-459c-499a-a18e-ebdfa13d7e26/volumes"
Nov 01 00:56:30.671862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.671836    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:30.678858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.678560    4493 generic.go:356] "Generic (PLEG): container finished" podID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerID="10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74" exitCode=0
Nov 01 00:56:30.678858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.678771    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" event={"ID":"703227a1-8bce-4dc7-a355-bb79f00312d8","Type":"ContainerDied","Data":"10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74"}
Nov 01 00:56:30.944250 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:30.944225    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:56:31.062659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.062619    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.062830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.062740    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.062830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.062783    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063012 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.062990    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063097 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063059    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063097    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063109    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063217    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063256    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063396    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063403    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063503    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\") pod \"703227a1-8bce-4dc7-a355-bb79f00312d8\" (UID: \"703227a1-8bce-4dc7-a355-bb79f00312d8\") "
Nov 01 00:56:31.063862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.063577    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:31.064330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.064276    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703227a1-8bce-4dc7-a355-bb79f00312d8-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.064330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.064324    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.064431 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.064347    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.064488 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.064415    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.064811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.064775    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:31.065584 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.065556    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:31.066127 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.066098    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:31.066271 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.066169    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "kube-api-access-g2r25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:31.067059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.067035    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo" pod "703227a1-8bce-4dc7-a355-bb79f00312d8" (UID: "703227a1-8bce-4dc7-a355-bb79f00312d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:56:31.104531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.104495    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 00:56:31.109340 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:31.109312    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57b67b9_dda2_46a4_a9b3_f38d49c18f13.slice/crio-0f1ac1660ff78a90c8b31ae1e96f540442a5fd0ca08e0d03fe272b465b999afd WatchSource:0}: Error finding container 0f1ac1660ff78a90c8b31ae1e96f540442a5fd0ca08e0d03fe272b465b999afd: Status 404 returned error can't find the container with id 0f1ac1660ff78a90c8b31ae1e96f540442a5fd0ca08e0d03fe272b465b999afd
Nov 01 00:56:31.166235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.166203    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.166235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.166227    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.166235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.166237    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-kube-api-access-g2r25\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.166447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.166246    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.166447 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.166254    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703227a1-8bce-4dc7-a355-bb79f00312d8-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:31.686939 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.686903    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" event={"ID":"b57b67b9-dda2-46a4-a9b3-f38d49c18f13","Type":"ContainerStarted","Data":"0f1ac1660ff78a90c8b31ae1e96f540442a5fd0ca08e0d03fe272b465b999afd"}
Nov 01 00:56:31.688708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.688681    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj" event={"ID":"703227a1-8bce-4dc7-a355-bb79f00312d8","Type":"ContainerDied","Data":"188cfcc3a651226aede78bff9f9d94e2f13d3c600d533a490889357aafab9930"}
Nov 01 00:56:31.688708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.688712    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"
Nov 01 00:56:31.688936 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.688736    4493 scope.go:122] "RemoveContainer" containerID="10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74"
Nov 01 00:56:31.875995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.875966    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:56:31.926126 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:31.926092    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-mncwj"]
Nov 01 00:56:32.504396 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.504365    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" path="/var/lib/kubelet/pods/703227a1-8bce-4dc7-a355-bb79f00312d8/volumes"
Nov 01 00:56:32.698715 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.698678    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" event={"ID":"b57b67b9-dda2-46a4-a9b3-f38d49c18f13","Type":"ContainerStarted","Data":"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"}
Nov 01 00:56:32.699147 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.698816    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:32.823896 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.823792    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" podStartSLOduration=2.858772814 podStartE2EDuration="3.823775288s" totalImagesPullingTime="965.002474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:32.819302756 +0000 UTC m=+1982.496889699" watchObservedRunningTime="2026-11-01 00:56:32.823775288 +0000 UTC m=+1982.501362233"
Nov 01 00:56:32.935309 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.935276    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:56:32.935540 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:32.935516    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy" containerID="cri-o://a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07" gracePeriod=30
Nov 01 00:56:35.257204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.257167    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:56:35.257727 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.257374    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" podUID="645a6b9f-8446-454c-aa57-2fb62d6896d4" containerName="maas-api" containerID="cri-o://986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890" gracePeriod=30
Nov 01 00:56:35.682159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.676158    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:56:35.832159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.830720    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\") pod \"645a6b9f-8446-454c-aa57-2fb62d6896d4\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") "
Nov 01 00:56:35.832159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.831973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\") pod \"645a6b9f-8446-454c-aa57-2fb62d6896d4\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") "
Nov 01 00:56:35.840217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.832411    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca" pod "645a6b9f-8446-454c-aa57-2fb62d6896d4" (UID: "645a6b9f-8446-454c-aa57-2fb62d6896d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:35.840217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.832937    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\" (UniqueName: \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\") pod \"645a6b9f-8446-454c-aa57-2fb62d6896d4\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") "
Nov 01 00:56:35.840217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.833344    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\") pod \"645a6b9f-8446-454c-aa57-2fb62d6896d4\" (UID: \"645a6b9f-8446-454c-aa57-2fb62d6896d4\") "
Nov 01 00:56:35.840217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.834776    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a6b9f-8446-454c-aa57-2fb62d6896d4-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:35.840217 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.837509    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs" pod "645a6b9f-8446-454c-aa57-2fb62d6896d4" (UID: "645a6b9f-8446-454c-aa57-2fb62d6896d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:56:35.862029 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.861975    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls" pod "645a6b9f-8446-454c-aa57-2fb62d6896d4" (UID: "645a6b9f-8446-454c-aa57-2fb62d6896d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:56:35.885270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.885208    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k" pod "645a6b9f-8446-454c-aa57-2fb62d6896d4" (UID: "645a6b9f-8446-454c-aa57-2fb62d6896d4"). InnerVolumeSpecName "kube-api-access-dcr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:35.954721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.954628    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcr5k\" (UniqueName: \"kubernetes.io/projected/645a6b9f-8446-454c-aa57-2fb62d6896d4-kube-api-access-dcr5k\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:35.954721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.954664    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:35.954721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.954679    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/645a6b9f-8446-454c-aa57-2fb62d6896d4-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:35.979003 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.978956    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"
Nov 01 00:56:35.979260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.979030    4493 generic.go:356] "Generic (PLEG): container finished" podID="645a6b9f-8446-454c-aa57-2fb62d6896d4" containerID="986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890" exitCode=0
Nov 01 00:56:35.979260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.979078    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" event={"ID":"645a6b9f-8446-454c-aa57-2fb62d6896d4","Type":"ContainerDied","Data":"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"}
Nov 01 00:56:35.979260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.979163    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp" event={"ID":"645a6b9f-8446-454c-aa57-2fb62d6896d4","Type":"ContainerDied","Data":"0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f"}
Nov 01 00:56:35.979260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:35.979196    4493 scope.go:122] "RemoveContainer" containerID="986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"
Nov 01 00:56:36.000183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:36.000111    4493 scope.go:122] "RemoveContainer" containerID="986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"
Nov 01 00:56:36.000699 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:36.000543    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890\": container with ID starting with 986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890 not found: ID does not exist" containerID="986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"
Nov 01 00:56:36.000699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:36.000576    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890"} err="failed to get container status \"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890\": rpc error: code = NotFound desc = could not find container \"986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890\": container with ID starting with 986211ee7c13fe88a8f257d63872b308690c5112f85eb2427d58f8e56d460890 not found: ID does not exist"
Nov 01 00:56:36.037392 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:36.037352    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645a6b9f_8446_454c_aa57_2fb62d6896d4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645a6b9f_8446_454c_aa57_2fb62d6896d4.slice/crio-0894b83694c0155565ee0ac9a6625597c040cfd0203eeaa308a0e22310d76a2f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:56:36.129004 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:36.128922    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:56:36.150580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:36.150548    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:56:36.154213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:36.154192    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f10727e-79779df49f-jlxdp"]
Nov 01 00:56:36.535770 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:36.535742    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645a6b9f-8446-454c-aa57-2fb62d6896d4" path="/var/lib/kubelet/pods/645a6b9f-8446-454c-aa57-2fb62d6896d4/volumes"
Nov 01 00:56:38.351382 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.351354    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:56:38.497261 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497228    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497287    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497332    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497399    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497449 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497425    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497456    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497659 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497638    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:38.497819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497675    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497770    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\") pod \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\" (UID: \"fb1469a4-3e0a-45f4-b1c2-26674471fe4e\") "
Nov 01 00:56:38.497819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497777    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:38.497965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497879    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:38.497965 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.497911    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:56:38.498260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498219    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:38.498767 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498742    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.498887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498772    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.498887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498792    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.498887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498805    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.498887 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.498819    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.499874 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.499837    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:56:38.500083 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.500060    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:56:38.500532 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.500507    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "kube-api-access-2qwmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:38.500627 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.500556    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token" pod "fb1469a4-3e0a-45f4-b1c2-26674471fe4e" (UID: "fb1469a4-3e0a-45f4-b1c2-26674471fe4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:56:38.600756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.600724    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.600756 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.600755    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.600955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.600766    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.600955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.600779    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwmx\" (UniqueName: \"kubernetes.io/projected/fb1469a4-3e0a-45f4-b1c2-26674471fe4e-kube-api-access-2qwmx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:56:38.712419 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:38.712389    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 00:56:39.174270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.174237    4493 generic.go:356] "Generic (PLEG): container finished" podID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerID="a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07" exitCode=0
Nov 01 00:56:39.174798 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.174770    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerDied","Data":"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"}
Nov 01 00:56:39.175028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.175010    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9" event={"ID":"fb1469a4-3e0a-45f4-b1c2-26674471fe4e","Type":"ContainerDied","Data":"29849ff9d2de1814856a5b1fc6c7918d8e5fb5cc0652e9f1b1b7dad2e81011e1"}
Nov 01 00:56:39.175208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.175190    4493 scope.go:122] "RemoveContainer" containerID="a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"
Nov 01 00:56:39.175467 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.175439    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"
Nov 01 00:56:39.359670 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.359638    4493 scope.go:122] "RemoveContainer" containerID="bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"
Nov 01 00:56:39.449231 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.430380    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:56:39.485565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.484929    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-13f6ab63-openshift-default-5484496857-54zk9"]
Nov 01 00:56:39.486037 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.486009    4493 scope.go:122] "RemoveContainer" containerID="a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"
Nov 01 00:56:39.486669 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:39.486647    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07\": container with ID starting with a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07 not found: ID does not exist" containerID="a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"
Nov 01 00:56:39.486891 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.486866    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07"} err="failed to get container status \"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07\": rpc error: code = NotFound desc = could not find container \"a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07\": container with ID starting with a5d661bfcc365472637c181b655aaaa7d307d330125720dafd63e8a9f67ebb07 not found: ID does not exist"
Nov 01 00:56:39.487000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.486989    4493 scope.go:122] "RemoveContainer" containerID="bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"
Nov 01 00:56:39.487415 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:39.487393    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870\": container with ID starting with bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870 not found: ID does not exist" containerID="bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"
Nov 01 00:56:39.487552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.487530    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870"} err="failed to get container status \"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870\": rpc error: code = NotFound desc = could not find container \"bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870\": container with ID starting with bf2043b1c43486a27f555a4a096cb236564c1e84bed46b84dd0507a24aff8870 not found: ID does not exist"
Nov 01 00:56:39.506022 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.505985    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 00:56:39.507296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507273    4493 cpu_manager.go:414] "containerMap: removing container" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy"
Nov 01 00:56:39.507452 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507438    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy"
Nov 01 00:56:39.507575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507565    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507646 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507637    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507713    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507729    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507737    4493 cpu_manager.go:414] "containerMap: removing container" podUID="645a6b9f-8446-454c-aa57-2fb62d6896d4" containerName="maas-api"
Nov 01 00:56:39.507877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507744    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a6b9f-8446-454c-aa57-2fb62d6896d4" containerName="maas-api"
Nov 01 00:56:39.507953 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507940    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="703227a1-8bce-4dc7-a355-bb79f00312d8" containerName="istio-proxy"
Nov 01 00:56:39.507987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507964    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507974    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" containerName="istio-proxy"
Nov 01 00:56:39.507987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.507982    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a6b9f-8446-454c-aa57-2fb62d6896d4" containerName="maas-api"
Nov 01 00:56:39.514014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.513990    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.564318 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.564289    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 00:56:39.576118 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.576093    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af82f1-openshift-default-dockercfg-bgsqw\""
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.662801    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5lq\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.662906    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.662970    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663013    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663064    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663153    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663370    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663405    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.664677 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.663443    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.768735 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.765190    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.768940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.768870    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.768940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.768924    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.768998    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769299 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769318    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769357    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769538 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769519    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5lq\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769608    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.769945 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769924    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.770085 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.769968    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.770233 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.770213    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.770940 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.770903    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.771268 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.771219    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.823987 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.823909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.833746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.833710    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.844202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.843593    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5lq\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:39.844202 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:39.844082    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\") pod \"e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") " pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:40.134212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.130576    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:40.505947 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.505917    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1469a4-3e0a-45f4-b1c2-26674471fe4e" path="/var/lib/kubelet/pods/fb1469a4-3e0a-45f4-b1c2-26674471fe4e/volumes"
Nov 01 00:56:40.686977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.686938    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 00:56:40.695847 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:40.695808    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3320394e_5d9c_43a9_a2f2_c847ed77f3c5.slice/crio-59913fd4ddd6ea384d4989ecb73314f2aa259225aff59a7ab09241f4ab05b727 WatchSource:0}: Error finding container 59913fd4ddd6ea384d4989ecb73314f2aa259225aff59a7ab09241f4ab05b727: Status 404 returned error can't find the container with id 59913fd4ddd6ea384d4989ecb73314f2aa259225aff59a7ab09241f4ab05b727
Nov 01 00:56:40.699277 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.699245    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:56:40.699373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.699311    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:56:40.699373 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:40.699350    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:56:41.265185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:41.265127    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerStarted","Data":"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"}
Nov 01 00:56:41.265340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:41.265204    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerStarted","Data":"59913fd4ddd6ea384d4989ecb73314f2aa259225aff59a7ab09241f4ab05b727"}
Nov 01 00:56:41.440579 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:41.440526    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podStartSLOduration=2.440512093 podStartE2EDuration="2.440512093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:41.43664383 +0000 UTC m=+1991.114230774" watchObservedRunningTime="2026-11-01 00:56:41.440512093 +0000 UTC m=+1991.118099038"
Nov 01 00:56:42.132571 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:42.132535    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:42.138349 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:42.138325    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:42.272365 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:42.272320    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:42.273214 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:42.273191    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:56:43.460192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.453845    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 00:56:43.555485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.555451    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:43.587602 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.587505    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 00:56:43.618159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.605206    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 00:56:43.674161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.624215    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-27dee6-dockercfg-qq2qk\""
Nov 01 00:56:43.680976 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.633377    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-27dee6\""
Nov 01 00:56:43.889168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.858685    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:43.889168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.870651    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mmc\" (UniqueName: \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:43.889168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.870878    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:43.915548 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.915494    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 00:56:43.973177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.972679    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q989x\" (UniqueName: \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:43.973177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.972719    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mmc\" (UniqueName: \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:43.973177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.972830    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:43.973177 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.972878    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:43.973932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:43.973905    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:44.034982 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.034945    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mmc\" (UniqueName: \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\") pod \"payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:44.074109 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.074078    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:44.074392 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.074375    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q989x\" (UniqueName: \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:44.074986 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.074962    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:44.135346 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.135302    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q989x\" (UniqueName: \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\") pod \"payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:44.222655 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.222610    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:44.332168 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.332094    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:44.612624 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:44.612594    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb088c408_387d_4f89_b320_ec70e5754808.slice/crio-6f653cb9271a41cf612526d9b36e6e510e17719f585f10eeb40727ca81e0cd2b WatchSource:0}: Error finding container 6f653cb9271a41cf612526d9b36e6e510e17719f585f10eeb40727ca81e0cd2b: Status 404 returned error can't find the container with id 6f653cb9271a41cf612526d9b36e6e510e17719f585f10eeb40727ca81e0cd2b
Nov 01 00:56:44.612624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.612602    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 00:56:44.720086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:44.720055    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 00:56:44.724190 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:44.724161    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70360ad_5b31_4781_8603_2332f23c9d6c.slice/crio-253f61c7873b11371d464ff136dedae89bc582abed4f01a194be23de016ca5a7 WatchSource:0}: Error finding container 253f61c7873b11371d464ff136dedae89bc582abed4f01a194be23de016ca5a7: Status 404 returned error can't find the container with id 253f61c7873b11371d464ff136dedae89bc582abed4f01a194be23de016ca5a7
Nov 01 00:56:44.991778 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:44.991740    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:56:45.371059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.370953    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" event={"ID":"d70360ad-5b31-4781-8603-2332f23c9d6c","Type":"ContainerStarted","Data":"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"}
Nov 01 00:56:45.371059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.371027    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:45.371059 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.371053    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" event={"ID":"d70360ad-5b31-4781-8603-2332f23c9d6c","Type":"ContainerStarted","Data":"253f61c7873b11371d464ff136dedae89bc582abed4f01a194be23de016ca5a7"}
Nov 01 00:56:45.372652 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.372627    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" event={"ID":"b088c408-387d-4f89-b320-ec70e5754808","Type":"ContainerStarted","Data":"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"}
Nov 01 00:56:45.372761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.372664    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" event={"ID":"b088c408-387d-4f89-b320-ec70e5754808","Type":"ContainerStarted","Data":"6f653cb9271a41cf612526d9b36e6e510e17719f585f10eeb40727ca81e0cd2b"}
Nov 01 00:56:45.372761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.372752    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:56:45.488200 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.488123    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" podStartSLOduration=2.488110296 podStartE2EDuration="2.488110296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:45.486663066 +0000 UTC m=+1995.164250023" watchObservedRunningTime="2026-11-01 00:56:45.488110296 +0000 UTC m=+1995.165697241"
Nov 01 00:56:45.617174 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:45.615496    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" podStartSLOduration=2.6154743700000003 podStartE2EDuration="2.61547437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:45.613603339 +0000 UTC m=+1995.291190286" watchObservedRunningTime="2026-11-01 00:56:45.61547437 +0000 UTC m=+1995.293061318"
Nov 01 00:56:46.213586 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:46.213548    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:56:48.914575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:48.914541    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 00:56:48.920673 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:48.920646    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 00:56:48.920811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:48.920764    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.006437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.006392    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.006633 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.006480    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvw7q\" (UniqueName: \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.023469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.023443    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-af82f1-dockercfg-kf5wm\""
Nov 01 00:56:49.023469 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.023447    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-af82f1\""
Nov 01 00:56:49.107685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.107649    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.107858 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.107709    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvw7q\" (UniqueName: \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.108414 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.108396    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.169510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.169484    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvw7q\" (UniqueName: \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\") pod \"payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.236405 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.236370    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.620213 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:56:49.620186    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fab8f56_4cb0_47ab_bcf6_48e501d42f93.slice/crio-72a8157b01c95d84b3cd6e8c46d497c8bc0662faa185cb99e45c4541788cff71 WatchSource:0}: Error finding container 72a8157b01c95d84b3cd6e8c46d497c8bc0662faa185cb99e45c4541788cff71: Status 404 returned error can't find the container with id 72a8157b01c95d84b3cd6e8c46d497c8bc0662faa185cb99e45c4541788cff71
Nov 01 00:56:49.620411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.620390    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 00:56:49.933622 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.933531    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" event={"ID":"3fab8f56-4cb0-47ab-bcf6-48e501d42f93","Type":"ContainerStarted","Data":"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"}
Nov 01 00:56:49.934011 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.933638    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:56:49.934011 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:49.933672    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" event={"ID":"3fab8f56-4cb0-47ab-bcf6-48e501d42f93","Type":"ContainerStarted","Data":"72a8157b01c95d84b3cd6e8c46d497c8bc0662faa185cb99e45c4541788cff71"}
Nov 01 00:56:50.051322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:50.051263    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" podStartSLOduration=2.051245568 podStartE2EDuration="2.051245568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:50.049330757 +0000 UTC m=+1999.726917701" watchObservedRunningTime="2026-11-01 00:56:50.051245568 +0000 UTC m=+1999.728832514"
Nov 01 00:56:50.887922 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:50.887886    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:56:56.293317 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:56:56.293275    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:56:56.383790 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:56.383761    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 00:56:56.383959 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:56:56.383826    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 00:57:00.976159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:00.960367    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 00:57:01.807912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:01.807866    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:57:01.820525 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:01.820480    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:57:01.921283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:01.921236    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 00:57:01.944345 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:01.944294    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:57:02.094340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:02.094226    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:57:05.876548 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:05.876502    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:06.322525 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:06.322483    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:06.864688 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:06.864642    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:06.864898 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:06.864766    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:09.894414 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.894373    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:57:09.894937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.894629    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" containerID="cri-o://80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d" gracePeriod=30
Nov 01 00:57:09.944321 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.944286    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:57:09.944554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.944530    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy" containerID="cri-o://968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b" gracePeriod=30
Nov 01 00:57:09.948234 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.948206    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:57:09.948443 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.948422    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy" containerID="cri-o://633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d" gracePeriod=30
Nov 01 00:57:09.995126 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.995089    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:57:09.995451 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:09.995424    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy" containerID="cri-o://7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7" gracePeriod=30
Nov 01 00:57:14.700084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:14.699998    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:57:14.700459 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:14.700070    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:57:14.995696 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:14.995665    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:57:15.277911 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.276726    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:57:15.290821 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.290793    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.290978 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.290960    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291041 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291016    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291100 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291057    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291287 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291201    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291287 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291274    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291324    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291371    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.291463    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\") pod \"1b2391d5-b464-416c-860a-6eee28e98be7\" (UID: \"1b2391d5-b464-416c-860a-6eee28e98be7\") "
Nov 01 00:57:15.292578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292024    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.292578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292294    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.292578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292453    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.292578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292466    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.292745 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292638    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:15.292781 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292743    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.292853 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.292825    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.294671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.294647    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:57:15.296724 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.296686    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "kube-api-access-njs65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.296995 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.296971    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.297564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.297126    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token" pod "1b2391d5-b464-416c-860a-6eee28e98be7" (UID: "1b2391d5-b464-416c-860a-6eee28e98be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.344174 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.343347    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:57:15.391671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.391643    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:57:15.393028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393008    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393146 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393042    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393146 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393066    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393173    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393202    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393232    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393426 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393293    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393426 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393310    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393426 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393376    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\") pod \"cd396d4d-601b-45cb-80b0-cd29ab967520\" (UID: \"cd396d4d-601b-45cb-80b0-cd29ab967520\") "
Nov 01 00:57:15.393580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393424    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.393580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393455    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.393580 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393510    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:15.393734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393652    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.393734 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.393674    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.394065 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394044    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394068    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394082    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njs65\" (UniqueName: \"kubernetes.io/projected/1b2391d5-b464-416c-860a-6eee28e98be7-kube-api-access-njs65\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394096    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd396d4d-601b-45cb-80b0-cd29ab967520-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394110    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394121    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394179    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2391d5-b464-416c-860a-6eee28e98be7-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394189    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394197    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394205    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2391d5-b464-416c-860a-6eee28e98be7-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394215    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.394375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.394224    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2391d5-b464-416c-860a-6eee28e98be7-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.395251 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.395231    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:57:15.395484 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.395467    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.395900 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.395883    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:57:15.396192 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.396176    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "kube-api-access-bh8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.396265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.396206    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy" pod "cd396d4d-601b-45cb-80b0-cd29ab967520" (UID: "cd396d4d-601b-45cb-80b0-cd29ab967520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.439884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.439851    4493 generic.go:356] "Generic (PLEG): container finished" podID="1b2391d5-b464-416c-860a-6eee28e98be7" containerID="80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d" exitCode=0
Nov 01 00:57:15.440053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.439965    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"
Nov 01 00:57:15.440053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.439992    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" event={"ID":"1b2391d5-b464-416c-860a-6eee28e98be7","Type":"ContainerDied","Data":"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"}
Nov 01 00:57:15.440053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.440043    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl" event={"ID":"1b2391d5-b464-416c-860a-6eee28e98be7","Type":"ContainerDied","Data":"b66c532c8678a60b0104697316862a719892073989a2900754648eb7cd4f470a"}
Nov 01 00:57:15.440265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.440067    4493 scope.go:122] "RemoveContainer" containerID="80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"
Nov 01 00:57:15.441705 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.441678    4493 generic.go:356] "Generic (PLEG): container finished" podID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerID="633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d" exitCode=0
Nov 01 00:57:15.441796 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.441766    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" event={"ID":"cd396d4d-601b-45cb-80b0-cd29ab967520","Type":"ContainerDied","Data":"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"}
Nov 01 00:57:15.441846 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.441808    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5" event={"ID":"cd396d4d-601b-45cb-80b0-cd29ab967520","Type":"ContainerDied","Data":"c581bb16b141f01832adb19e4a9415c6526d9fc2b739ac686f1f0b51b1ac0adc"}
Nov 01 00:57:15.442207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.441895    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"
Nov 01 00:57:15.443510 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.443489    4493 generic.go:356] "Generic (PLEG): container finished" podID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerID="968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b" exitCode=0
Nov 01 00:57:15.443680 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.443664    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" event={"ID":"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be","Type":"ContainerDied","Data":"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"}
Nov 01 00:57:15.443726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.443707    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2" event={"ID":"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be","Type":"ContainerDied","Data":"d9c1b52e1096e2c10bdfa8e84531e6757eac503d14c0451297c665c27a91b2d7"}
Nov 01 00:57:15.443784 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.443772    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"
Nov 01 00:57:15.447161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.447110    4493 generic.go:356] "Generic (PLEG): container finished" podID="55739022-d29a-4b36-ad08-001f97401bd9" containerID="7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7" exitCode=0
Nov 01 00:57:15.447280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.447171    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerDied","Data":"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"}
Nov 01 00:57:15.447280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.447185    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"
Nov 01 00:57:15.447280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.447222    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h" event={"ID":"55739022-d29a-4b36-ad08-001f97401bd9","Type":"ContainerDied","Data":"cf2174d4a08f33e882272680610d1b47fee030378a1d26a4a9bb8f48df3f4731"}
Nov 01 00:57:15.453560 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.453545    4493 scope.go:122] "RemoveContainer" containerID="80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"
Nov 01 00:57:15.453813 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:15.453797    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d\": container with ID starting with 80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d not found: ID does not exist" containerID="80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"
Nov 01 00:57:15.453854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.453820    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d"} err="failed to get container status \"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d\": rpc error: code = NotFound desc = could not find container \"80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d\": container with ID starting with 80c699310f80d7d5296bd548d596bb65dd5e352001c9d5d7b3b4b9afaba5d46d not found: ID does not exist"
Nov 01 00:57:15.453854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.453835    4493 scope.go:122] "RemoveContainer" containerID="633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"
Nov 01 00:57:15.464199 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.464180    4493 scope.go:122] "RemoveContainer" containerID="633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"
Nov 01 00:57:15.464474 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:15.464448    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d\": container with ID starting with 633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d not found: ID does not exist" containerID="633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"
Nov 01 00:57:15.464574 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.464479    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d"} err="failed to get container status \"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d\": rpc error: code = NotFound desc = could not find container \"633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d\": container with ID starting with 633ba1cda41803c58e3888a6e022d3beda4c649c575020c9b1c71f476f538e8d not found: ID does not exist"
Nov 01 00:57:15.464574 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.464504    4493 scope.go:122] "RemoveContainer" containerID="968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"
Nov 01 00:57:15.474999 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.474977    4493 scope.go:122] "RemoveContainer" containerID="968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"
Nov 01 00:57:15.475413 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:15.475390    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b\": container with ID starting with 968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b not found: ID does not exist" containerID="968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"
Nov 01 00:57:15.475535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.475419    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b"} err="failed to get container status \"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b\": rpc error: code = NotFound desc = could not find container \"968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b\": container with ID starting with 968228480ea4473e280a5f425dc020af37900810c695b685fb73a6e774683f6b not found: ID does not exist"
Nov 01 00:57:15.475535 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.475439    4493 scope.go:122] "RemoveContainer" containerID="7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"
Nov 01 00:57:15.486784 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.486764    4493 scope.go:122] "RemoveContainer" containerID="0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"
Nov 01 00:57:15.494813 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494785    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.494927 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494819    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.494927 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494862    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.494927 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494894    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494934    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.494973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495005    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495114 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495101    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495177    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495219    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495341    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495432    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495510    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495549 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495546    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495581    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495612    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.495850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495717    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\") pod \"55739022-d29a-4b36-ad08-001f97401bd9\" (UID: \"55739022-d29a-4b36-ad08-001f97401bd9\") "
Nov 01 00:57:15.495850 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495750    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\") pod \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\" (UID: \"4530a76e-1ff2-4c6b-b8d4-5a8bed5766be\") "
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495431    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496332    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496522    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.495563    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496236    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496263    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496560    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496754    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.496902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.496797    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497158    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497181    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55739022-d29a-4b36-ad08-001f97401bd9-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497197    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497211    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497224    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497238    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497251    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497264    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497277    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497291    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8wn\" (UniqueName: \"kubernetes.io/projected/cd396d4d-601b-45cb-80b0-cd29ab967520-kube-api-access-bh8wn\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497306    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497320    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd396d4d-601b-45cb-80b0-cd29ab967520-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497333    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.497748 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.497348    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.498877 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.498619    4493 scope.go:122] "RemoveContainer" containerID="7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"
Nov 01 00:57:15.499247 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:15.499069    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7\": container with ID starting with 7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7 not found: ID does not exist" containerID="7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"
Nov 01 00:57:15.499247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499108    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7"} err="failed to get container status \"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7\": rpc error: code = NotFound desc = could not find container \"7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7\": container with ID starting with 7735d3b5373e3edc907f06cfc5963a55daee80fed522e2243faffaebec56fad7 not found: ID does not exist"
Nov 01 00:57:15.499247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499169    4493 scope.go:122] "RemoveContainer" containerID="0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"
Nov 01 00:57:15.499785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499636    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "kube-api-access-98nmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.499785 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:15.499725    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c\": container with ID starting with 0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c not found: ID does not exist" containerID="0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"
Nov 01 00:57:15.499785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499745    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "kube-api-access-l9wb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.499785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499756    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c"} err="failed to get container status \"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c\": rpc error: code = NotFound desc = could not find container \"0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c\": container with ID starting with 0f8ee613df5f1a7367b202dc226fed0ec0a667157c99d8d819870b997c8d965c not found: ID does not exist"
Nov 01 00:57:15.500049 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.499942    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.500393 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.500371    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.501487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.501438    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:57:15.501626 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.501603    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:15.501821 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.501799    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo" pod "55739022-d29a-4b36-ad08-001f97401bd9" (UID: "55739022-d29a-4b36-ad08-001f97401bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:57:15.502688 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.502667    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo" pod "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" (UID: "4530a76e-1ff2-4c6b-b8d4-5a8bed5766be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:57:15.580455 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.580418    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:57:15.584257 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.584226    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-gsssl"]
Nov 01 00:57:15.598907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598879    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nmc\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-kube-api-access-98nmc\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.598907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598904    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55739022-d29a-4b36-ad08-001f97401bd9-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598913    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598923    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55739022-d29a-4b36-ad08-001f97401bd9-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598932    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598940    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598949    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wb5\" (UniqueName: \"kubernetes.io/projected/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be-kube-api-access-l9wb5\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:15.599069 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:15.598957    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55739022-d29a-4b36-ad08-001f97401bd9-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:16.015105 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.015018    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:57:16.019639 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.019613    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-9r2x5"]
Nov 01 00:57:16.135933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.135888    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:57:16.140322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.140295    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-vff6h"]
Nov 01 00:57:16.249859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.249827    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:57:16.255870 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.255843    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-wbgd2"]
Nov 01 00:57:16.402932 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:16.402852    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:16.506117 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.506086    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" path="/var/lib/kubelet/pods/1b2391d5-b464-416c-860a-6eee28e98be7/volumes"
Nov 01 00:57:16.506662 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.506648    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" path="/var/lib/kubelet/pods/4530a76e-1ff2-4c6b-b8d4-5a8bed5766be/volumes"
Nov 01 00:57:16.507081 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.507069    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55739022-d29a-4b36-ad08-001f97401bd9" path="/var/lib/kubelet/pods/55739022-d29a-4b36-ad08-001f97401bd9/volumes"
Nov 01 00:57:16.507890 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:16.507875    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" path="/var/lib/kubelet/pods/cd396d4d-601b-45cb-80b0-cd29ab967520/volumes"
Nov 01 00:57:18.745425 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:18.745390    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:57:18.851956 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:18.851923    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:57:20.887143 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:20.887089    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:23.745781 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:23.745743    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" podUID="b191a478-5321-4b31-9811-0c6c8b142483" containerName="payload-processing" containerID="cri-o://b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce" gracePeriod=25
Nov 01 00:57:23.852783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:23.852737    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" podUID="e3e61b05-6a17-4cf0-be39-65d688c3f926" containerName="payload-pre-processing" containerID="cri-o://ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76" gracePeriod=25
Nov 01 00:57:24.037578 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.037546    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:57:24.166764 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.166737    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:57:24.186014 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.185983    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\" (UniqueName: \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\") pod \"b191a478-5321-4b31-9811-0c6c8b142483\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") "
Nov 01 00:57:24.186283 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.186262    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\") pod \"b191a478-5321-4b31-9811-0c6c8b142483\" (UID: \"b191a478-5321-4b31-9811-0c6c8b142483\") "
Nov 01 00:57:24.186879 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.186673    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume" pod "b191a478-5321-4b31-9811-0c6c8b142483" (UID: "b191a478-5321-4b31-9811-0c6c8b142483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:24.187066 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.187042    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b191a478-5321-4b31-9811-0c6c8b142483-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:24.188362 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.188336    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx" pod "b191a478-5321-4b31-9811-0c6c8b142483" (UID: "b191a478-5321-4b31-9811-0c6c8b142483"). InnerVolumeSpecName "kube-api-access-jpwbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:24.287859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.287765    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\") pod \"e3e61b05-6a17-4cf0-be39-65d688c3f926\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") "
Nov 01 00:57:24.287859 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.287852    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\" (UniqueName: \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\") pod \"e3e61b05-6a17-4cf0-be39-65d688c3f926\" (UID: \"e3e61b05-6a17-4cf0-be39-65d688c3f926\") "
Nov 01 00:57:24.288183 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.288160    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume" pod "e3e61b05-6a17-4cf0-be39-65d688c3f926" (UID: "e3e61b05-6a17-4cf0-be39-65d688c3f926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:24.288596 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.288579    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwbx\" (UniqueName: \"kubernetes.io/projected/b191a478-5321-4b31-9811-0c6c8b142483-kube-api-access-jpwbx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:24.288634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.288600    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e61b05-6a17-4cf0-be39-65d688c3f926-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:24.290171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.290149    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh" pod "e3e61b05-6a17-4cf0-be39-65d688c3f926" (UID: "e3e61b05-6a17-4cf0-be39-65d688c3f926"). InnerVolumeSpecName "kube-api-access-pttnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:24.389955 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.389923    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttnh\" (UniqueName: \"kubernetes.io/projected/e3e61b05-6a17-4cf0-be39-65d688c3f926-kube-api-access-pttnh\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:24.520252 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.520226    4493 generic.go:356] "Generic (PLEG): container finished" podID="e3e61b05-6a17-4cf0-be39-65d688c3f926" containerID="ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76" exitCode=0
Nov 01 00:57:24.520398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.520257    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" event={"ID":"e3e61b05-6a17-4cf0-be39-65d688c3f926","Type":"ContainerDied","Data":"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"}
Nov 01 00:57:24.520398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.520283    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"
Nov 01 00:57:24.520398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.520327    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb" event={"ID":"e3e61b05-6a17-4cf0-be39-65d688c3f926","Type":"ContainerDied","Data":"7b06f6b07ba64c5d37353ea3676231a5f52c399e6f904bcfcf78064975de3709"}
Nov 01 00:57:24.520398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.520357    4493 scope.go:122] "RemoveContainer" containerID="ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"
Nov 01 00:57:24.522187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.522163    4493 generic.go:356] "Generic (PLEG): container finished" podID="b191a478-5321-4b31-9811-0c6c8b142483" containerID="b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce" exitCode=0
Nov 01 00:57:24.522296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.522199    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"
Nov 01 00:57:24.522296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.522215    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" event={"ID":"b191a478-5321-4b31-9811-0c6c8b142483","Type":"ContainerDied","Data":"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"}
Nov 01 00:57:24.522296 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.522256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p" event={"ID":"b191a478-5321-4b31-9811-0c6c8b142483","Type":"ContainerDied","Data":"21f8c39bc06887d8e3e879783c9551629924b0cda0d3aae2f798e1265f27ba23"}
Nov 01 00:57:24.531216 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.531192    4493 scope.go:122] "RemoveContainer" containerID="ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"
Nov 01 00:57:24.531519 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:24.531493    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76\": container with ID starting with ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76 not found: ID does not exist" containerID="ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"
Nov 01 00:57:24.531600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.531529    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76"} err="failed to get container status \"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76\": rpc error: code = NotFound desc = could not find container \"ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76\": container with ID starting with ed7e382b21387c78f92f8bd228ca8f5243c10107c6c4d97a2e51ea2275e5ef76 not found: ID does not exist"
Nov 01 00:57:24.531600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.531549    4493 scope.go:122] "RemoveContainer" containerID="b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"
Nov 01 00:57:24.540848 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.540826    4493 scope.go:122] "RemoveContainer" containerID="b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"
Nov 01 00:57:24.541123 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:24.541103    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce\": container with ID starting with b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce not found: ID does not exist" containerID="b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"
Nov 01 00:57:24.541291 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.541154    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce"} err="failed to get container status \"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce\": rpc error: code = NotFound desc = could not find container \"b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce\": container with ID starting with b010f6f20e5845647f07b0d08aac41815e0feeb0fcf4c1b76371ba71cf37c2ce not found: ID does not exist"
Nov 01 00:57:24.726862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.725178    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:57:24.758207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.754962    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a31bcdff-5c9db47488-96d9p"]
Nov 01 00:57:24.812242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.812104    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:57:24.816492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:24.816468    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a31bcdff-5ff9cd479f-jp4mb"]
Nov 01 00:57:26.480348 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:26.480315    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703227a1_8bce_4dc7_a355_bb79f00312d8.slice/crio-10f926ff3eef9ec43b5f4f449387c4b124640d59aeda8315e96acb1231be0e74.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 00:57:26.504421 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:26.504386    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b191a478-5321-4b31-9811-0c6c8b142483" path="/var/lib/kubelet/pods/b191a478-5321-4b31-9811-0c6c8b142483/volumes"
Nov 01 00:57:26.504975 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:26.504954    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e61b05-6a17-4cf0-be39-65d688c3f926" path="/var/lib/kubelet/pods/e3e61b05-6a17-4cf0-be39-65d688c3f926/volumes"
Nov 01 00:57:30.922506 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:30.922465    4493 scope.go:122] "RemoveContainer" containerID="9339e2d5567032f1d7ea66f2858a5da73990619f3e6a92ce0df05dbfa6177c48"
Nov 01 00:57:33.125157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:33.123798    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:57:44.990858 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:44.990823    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:57:45.397923 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.397840    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 00:57:45.398561 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398545    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398563    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398578    4493 cpu_manager.go:414] "containerMap: removing container" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398585    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398596    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e3e61b05-6a17-4cf0-be39-65d688c3f926" containerName="payload-pre-processing"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398602    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e61b05-6a17-4cf0-be39-65d688c3f926" containerName="payload-pre-processing"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398624    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398628    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398636    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b191a478-5321-4b31-9811-0c6c8b142483" containerName="payload-processing"
Nov 01 00:57:45.398649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398641    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b191a478-5321-4b31-9811-0c6c8b142483" containerName="payload-processing"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398658    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398662    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398762    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398770    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd396d4d-601b-45cb-80b0-cd29ab967520" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398778    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b191a478-5321-4b31-9811-0c6c8b142483" containerName="payload-processing"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398787    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4530a76e-1ff2-4c6b-b8d4-5a8bed5766be" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398796    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2391d5-b464-416c-860a-6eee28e98be7" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398801    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:57:45.398908 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.398806    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e61b05-6a17-4cf0-be39-65d688c3f926" containerName="payload-pre-processing"
Nov 01 00:57:45.404315 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.404298    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.458863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.458806    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-af82f1-serving-cert\""
Nov 01 00:57:45.463702 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.463672    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 00:57:45.570294 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.570259    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.570472 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.570441    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.570538 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.570508    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvtb\" (UniqueName: \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.570689 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.570665    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.672552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.672456    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvtb\" (UniqueName: \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.672731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.672557    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.672731 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.672598    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.672731 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:45.672700    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-serving-cert: secret "maas-api-e2e-shared-a-w4-af82f1-serving-cert" not found
Nov 01 00:57:45.672893 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:45.672768    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls podName:f8a4dde4-1f43-46d2-8411-eac45411bbf1 nodeName:}" failed. No retries permitted until 2026-11-01 00:57:46.172748941 +0000 UTC m=+2055.850335887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls") pod "maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" (UID: "f8a4dde4-1f43-46d2-8411-eac45411bbf1") : secret "maas-api-e2e-shared-a-w4-af82f1-serving-cert" not found
Nov 01 00:57:45.672893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.672837    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.673399 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.673377    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.675463 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.675437    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:45.750159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:45.749335    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvtb\" (UniqueName: \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:46.223241 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:46.223181    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:46.225822 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:46.225798    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:46.319752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:46.319663    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:47.006654 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:47.006628    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 00:57:47.008084 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:57:47.008057    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a4dde4_1f43_46d2_8411_eac45411bbf1.slice/crio-2514204776fc3b6830181c48bfa044a7f1c0d08fe2313978fcc2835b000a1b73 WatchSource:0}: Error finding container 2514204776fc3b6830181c48bfa044a7f1c0d08fe2313978fcc2835b000a1b73: Status 404 returned error can't find the container with id 2514204776fc3b6830181c48bfa044a7f1c0d08fe2313978fcc2835b000a1b73
Nov 01 00:57:47.009833 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:47.009813    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 00:57:47.877235 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:47.877191    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" event={"ID":"f8a4dde4-1f43-46d2-8411-eac45411bbf1","Type":"ContainerStarted","Data":"2514204776fc3b6830181c48bfa044a7f1c0d08fe2313978fcc2835b000a1b73"}
Nov 01 00:57:48.976159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:48.973340    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" event={"ID":"f8a4dde4-1f43-46d2-8411-eac45411bbf1","Type":"ContainerStarted","Data":"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"}
Nov 01 00:57:48.976159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:48.974581    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:49.107672 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:49.107623    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" podStartSLOduration=3.216587948 podStartE2EDuration="4.107607873s" totalImagesPullingTime="891.019925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:49.10309419 +0000 UTC m=+2058.780681147" watchObservedRunningTime="2026-11-01 00:57:49.107607873 +0000 UTC m=+2058.785194877"
Nov 01 00:57:55.479713 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:55.479651    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:57:55.503471 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:55.503435    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" podUID="51b6791a-8750-4f5b-aae3-68a92a372b00" containerName="maas-api" containerID="cri-o://54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e" gracePeriod=30
Nov 01 00:57:56.126212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.118615    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 00:57:56.249277 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.249253    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:57:56.260260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.259445    4493 generic.go:356] "Generic (PLEG): container finished" podID="51b6791a-8750-4f5b-aae3-68a92a372b00" containerID="54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e" exitCode=0
Nov 01 00:57:56.260260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.259573    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" event={"ID":"51b6791a-8750-4f5b-aae3-68a92a372b00","Type":"ContainerDied","Data":"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"}
Nov 01 00:57:56.260260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.259611    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf" event={"ID":"51b6791a-8750-4f5b-aae3-68a92a372b00","Type":"ContainerDied","Data":"cfe2c33ce38f3688d80d87d298eda2b38148c95caf02bcf2539184f61fcef722"}
Nov 01 00:57:56.260260 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.259632    4493 scope.go:122] "RemoveContainer" containerID="54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"
Nov 01 00:57:56.272111 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.272091    4493 scope.go:122] "RemoveContainer" containerID="54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"
Nov 01 00:57:56.272417 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:57:56.272397    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e\": container with ID starting with 54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e not found: ID does not exist" containerID="54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"
Nov 01 00:57:56.272466 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.272427    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e"} err="failed to get container status \"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e\": rpc error: code = NotFound desc = could not find container \"54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e\": container with ID starting with 54104e25d14d984cadfefc93d06019961ab83218e220be698390727d022a299e not found: ID does not exist"
Nov 01 00:57:56.301746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.301716    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\") pod \"51b6791a-8750-4f5b-aae3-68a92a372b00\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") "
Nov 01 00:57:56.301935 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.301897    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\" (UniqueName: \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\") pod \"51b6791a-8750-4f5b-aae3-68a92a372b00\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") "
Nov 01 00:57:56.302028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.301976    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") pod \"51b6791a-8750-4f5b-aae3-68a92a372b00\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") "
Nov 01 00:57:56.302088 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.302041    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\") pod \"51b6791a-8750-4f5b-aae3-68a92a372b00\" (UID: \"51b6791a-8750-4f5b-aae3-68a92a372b00\") "
Nov 01 00:57:56.302660 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.302630    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca" pod "51b6791a-8750-4f5b-aae3-68a92a372b00" (UID: "51b6791a-8750-4f5b-aae3-68a92a372b00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:57:56.302819 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.302801    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b6791a-8750-4f5b-aae3-68a92a372b00-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:56.304857 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.304826    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls" pod "51b6791a-8750-4f5b-aae3-68a92a372b00" (UID: "51b6791a-8750-4f5b-aae3-68a92a372b00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:57:56.305113 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.305045    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs" pod "51b6791a-8750-4f5b-aae3-68a92a372b00" (UID: "51b6791a-8750-4f5b-aae3-68a92a372b00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 00:57:56.306554 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.306524    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt" pod "51b6791a-8750-4f5b-aae3-68a92a372b00" (UID: "51b6791a-8750-4f5b-aae3-68a92a372b00"). InnerVolumeSpecName "kube-api-access-zlzgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:57:56.404634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.404562    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzgt\" (UniqueName: \"kubernetes.io/projected/51b6791a-8750-4f5b-aae3-68a92a372b00-kube-api-access-zlzgt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:56.404634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.404592    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:56.404634 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:56.404603    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b6791a-8750-4f5b-aae3-68a92a372b00-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:57:57.267487 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:57.267460    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"
Nov 01 00:57:57.436933 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:57.436902    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:57:57.450301 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:57.450268    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a31bcdff-5496c658b-587gf"]
Nov 01 00:57:58.504607 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:57:58.504573    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b6791a-8750-4f5b-aae3-68a92a372b00" path="/var/lib/kubelet/pods/51b6791a-8750-4f5b-aae3-68a92a372b00/volumes"
Nov 01 00:58:14.991092 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:14.991055    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:58:16.504157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.479180    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 00:58:16.510216 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.505867    4493 cpu_manager.go:414] "containerMap: removing container" podUID="51b6791a-8750-4f5b-aae3-68a92a372b00" containerName="maas-api"
Nov 01 00:58:16.510460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.510435    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b6791a-8750-4f5b-aae3-68a92a372b00" containerName="maas-api"
Nov 01 00:58:16.510600 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.510587    4493 cpu_manager.go:414] "containerMap: removing container" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:58:16.510685 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.510674    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55739022-d29a-4b36-ad08-001f97401bd9" containerName="istio-proxy"
Nov 01 00:58:16.511058 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.511037    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b6791a-8750-4f5b-aae3-68a92a372b00" containerName="maas-api"
Nov 01 00:58:16.552432 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.552394    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.586977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.586943    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 00:58:16.682161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.648773    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f97e89\""
Nov 01 00:58:16.682161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.648926    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f97e89-dockercfg-777b2\""
Nov 01 00:58:16.715163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.709539    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnf6x\" (UniqueName: \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.715163 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.709706    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.779207 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.765118    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 00:58:16.797699 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.797668    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 00:58:16.798049 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.798034    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:16.815187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.815126    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.817494 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.817464    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf6x\" (UniqueName: \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.824032 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.824001    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.904245 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.904205    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf6x\" (UniqueName: \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\") pod \"payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:16.918937 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.918899    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7d7z\" (UniqueName: \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:16.919167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:16.919086    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.020388 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.020354    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7d7z\" (UniqueName: \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.020574 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.020451    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.021303 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.021279    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.081810 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.081739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7d7z\" (UniqueName: \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\") pod \"payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") " pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.141647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.141619    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:17.203504 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.203473    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:17.524713 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.524686    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 00:58:17.525386 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:58:17.525358    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da4368c_309d_4db3_8f66_353a68198ee9.slice/crio-11c39f211b155a3ef0125ec190a51e1bf58f0345bb1e6ac5ddd0f9b5b61917ff WatchSource:0}: Error finding container 11c39f211b155a3ef0125ec190a51e1bf58f0345bb1e6ac5ddd0f9b5b61917ff: Status 404 returned error can't find the container with id 11c39f211b155a3ef0125ec190a51e1bf58f0345bb1e6ac5ddd0f9b5b61917ff
Nov 01 00:58:17.626785 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:17.626761    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 00:58:17.627891 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:58:17.627860    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21233329_fa85_43ee_a4e9_9f41ff989632.slice/crio-d4480752c7312b2242a70a43b2f644ec00e35b1e8bc08c86a869038fb3b013fb WatchSource:0}: Error finding container d4480752c7312b2242a70a43b2f644ec00e35b1e8bc08c86a869038fb3b013fb: Status 404 returned error can't find the container with id d4480752c7312b2242a70a43b2f644ec00e35b1e8bc08c86a869038fb3b013fb
Nov 01 00:58:18.487274 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.487236    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" event={"ID":"21233329-fa85-43ee-a4e9-9f41ff989632","Type":"ContainerStarted","Data":"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"}
Nov 01 00:58:18.487491 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.487290    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" event={"ID":"21233329-fa85-43ee-a4e9-9f41ff989632","Type":"ContainerStarted","Data":"d4480752c7312b2242a70a43b2f644ec00e35b1e8bc08c86a869038fb3b013fb"}
Nov 01 00:58:18.487491 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.487346    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:18.489340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.489311    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" event={"ID":"3da4368c-309d-4db3-8f66-353a68198ee9","Type":"ContainerStarted","Data":"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"}
Nov 01 00:58:18.489489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.489363    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" event={"ID":"3da4368c-309d-4db3-8f66-353a68198ee9","Type":"ContainerStarted","Data":"11c39f211b155a3ef0125ec190a51e1bf58f0345bb1e6ac5ddd0f9b5b61917ff"}
Nov 01 00:58:18.489489 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.489391    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:18.606708 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.606656    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" podStartSLOduration=2.606640869 podStartE2EDuration="2.606640869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:18.602757086 +0000 UTC m=+2088.280344032" watchObservedRunningTime="2026-11-01 00:58:18.606640869 +0000 UTC m=+2088.284227867"
Nov 01 00:58:18.722265 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:18.722217    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" podStartSLOduration=2.722203277 podStartE2EDuration="2.722203277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:18.717827685 +0000 UTC m=+2088.395414642" watchObservedRunningTime="2026-11-01 00:58:18.722203277 +0000 UTC m=+2088.399790221"
Nov 01 00:58:29.501210 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:29.501179    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 00:58:29.501669 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:29.501247    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 00:58:35.767049 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:35.767006    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 00:58:36.884165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:36.878538    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:58:36.884165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:36.878937    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy" containerID="cri-o://fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966" gracePeriod=30
Nov 01 00:58:37.742453 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.742416    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:58:37.742706 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.742677    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" containerID="cri-o://170702f1870773dc46f820970c78cd0791c254d6cbc2b84eb906946ab06be7b3" gracePeriod=30
Nov 01 00:58:37.752264 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.752228    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:58:37.762028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.752638    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:58:37.762028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.752914    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" containerID="cri-o://0d763b801e01851a07954de5ac994bb3df8904b46ffcc46da3c93aacf2f3ea4b" gracePeriod=30
Nov 01 00:58:37.762028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:37.757586    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" containerID="cri-o://f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576" gracePeriod=30
Nov 01 00:58:39.253872 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:39.253829    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:58:39.254624 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:39.254545    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:58:41.962807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:41.962713    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:58:41.963242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:41.962796    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:58:42.455703 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.455677    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:58:42.556792 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.556688    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.556977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.556812    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.556977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.556844    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.556977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.556920    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.556963    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557031    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557073    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557119    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557163    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557188 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557189    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557521 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557352    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\") pod \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\" (UID: \"bf82d413-4b6c-4871-8dc6-dfef0a86abfb\") "
Nov 01 00:58:42.557521 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557401    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:42.557521 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557415    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:58:42.557765 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.557745    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:42.558438 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.558415    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.558523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.558446    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.558523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.558460    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.558523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.558473    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.558523 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.558487    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.559723 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.559700    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:58:42.559818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.559737    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:42.559916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.559897    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "kube-api-access-sw6np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:42.559916 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.559899    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy" pod "bf82d413-4b6c-4871-8dc6-dfef0a86abfb" (UID: "bf82d413-4b6c-4871-8dc6-dfef0a86abfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:42.659893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.659849    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.659893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.659883    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.659893 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.659895    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6np\" (UniqueName: \"kubernetes.io/projected/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-kube-api-access-sw6np\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.660209 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.659905    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82d413-4b6c-4871-8dc6-dfef0a86abfb-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:42.714547 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.714514    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 00:58:42.738161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.715613    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy"
Nov 01 00:58:42.738161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.715633    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy"
Nov 01 00:58:42.738161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.715790    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerName="istio-proxy"
Nov 01 00:58:42.760783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.757909    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 00:58:42.760783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.758017    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 00:58:42.760783 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.759733    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:42.776428 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.776396    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 00:58:42.826185 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.813469    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-44c9bf-serving-cert\""
Nov 01 00:58:42.862839 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.862804    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:42.863479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.862892    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsp2\" (UniqueName: \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:42.863479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.862986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:42.863479 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.863098    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:42.902761 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.902725    4493 generic.go:356] "Generic (PLEG): container finished" podID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerID="170702f1870773dc46f820970c78cd0791c254d6cbc2b84eb906946ab06be7b3" exitCode=0
Nov 01 00:58:42.902931 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.902790    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" event={"ID":"c63079b0-5263-47db-9bb0-41eb3d415ca0","Type":"ContainerDied","Data":"170702f1870773dc46f820970c78cd0791c254d6cbc2b84eb906946ab06be7b3"}
Nov 01 00:58:42.904645 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.904624    4493 generic.go:356] "Generic (PLEG): container finished" podID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerID="0d763b801e01851a07954de5ac994bb3df8904b46ffcc46da3c93aacf2f3ea4b" exitCode=0
Nov 01 00:58:42.904795 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.904696    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" event={"ID":"61cf6c5e-cae8-4e69-b061-2244b8732a49","Type":"ContainerDied","Data":"0d763b801e01851a07954de5ac994bb3df8904b46ffcc46da3c93aacf2f3ea4b"}
Nov 01 00:58:42.906001 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.905981    4493 generic.go:356] "Generic (PLEG): container finished" podID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" containerID="fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966" exitCode=0
Nov 01 00:58:42.906106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.906035    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"
Nov 01 00:58:42.906106 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.906081    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" event={"ID":"bf82d413-4b6c-4871-8dc6-dfef0a86abfb","Type":"ContainerDied","Data":"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"}
Nov 01 00:58:42.906281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.906156    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f" event={"ID":"bf82d413-4b6c-4871-8dc6-dfef0a86abfb","Type":"ContainerDied","Data":"42186937289cba1ed0f20d3b5055e3b3539200d014869f068632ffe4d14b7a38"}
Nov 01 00:58:42.906281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.906182    4493 scope.go:122] "RemoveContainer" containerID="fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"
Nov 01 00:58:42.956323 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.954568    4493 scope.go:122] "RemoveContainer" containerID="fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"
Nov 01 00:58:42.956323 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:42.954990    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966\": container with ID starting with fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966 not found: ID does not exist" containerID="fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"
Nov 01 00:58:42.956323 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:42.955026    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966"} err="failed to get container status \"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966\": rpc error: code = NotFound desc = could not find container \"fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966\": container with ID starting with fa2c4f144e7b5a6db421b7b138748c751e364a84a3a6cee338019b9867ddd966 not found: ID does not exist"
Nov 01 00:58:43.032071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.031021    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.032071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.031346    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.032071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.031420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsp2\" (UniqueName: \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.032071 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.031521    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.040181 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.034423    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.114671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.092794    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.114671 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.094693    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.192954 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.192228    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:58:43.195804 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.195770    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsp2\" (UniqueName: \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\") pod \"maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.275174 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.275112    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-mlf4f"]
Nov 01 00:58:43.365860 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.365797    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:58:43.380466 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.380418    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552006    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552122    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552204    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552333    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552410    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552473    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552638    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.553220 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.552671    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\") pod \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\" (UID: \"e26aaa66-5506-4b70-b8ad-b2e77d633aeb\") "
Nov 01 00:58:43.556700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.556326    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:58:43.556700 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.556625    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.557228 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.557062    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.557435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.557385    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.557530 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.557468    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.558499 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.557817    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:58:43.560178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.559713    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.560842 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.560819    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "kube-api-access-782dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.561649 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.561491    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token" pod "e26aaa66-5506-4b70-b8ad-b2e77d633aeb" (UID: "e26aaa66-5506-4b70-b8ad-b2e77d633aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.565473 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.565451    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656739    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656773    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656791    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656804    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656817    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-782dq\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-kube-api-access-782dq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656832    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656845    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656858    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.672587 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.656871    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26aaa66-5506-4b70-b8ad-b2e77d633aeb-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.688892 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.688864    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:58:43.757398 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757364    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757601 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757541    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757601 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757584    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757678    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757712 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757682    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.757810 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757777    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757865 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757841    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.757917 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757847    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757974 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757915    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.757974 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757962    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.758078 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.757998    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\") pod \"c63079b0-5263-47db-9bb0-41eb3d415ca0\" (UID: \"c63079b0-5263-47db-9bb0-41eb3d415ca0\") "
Nov 01 00:58:43.758389 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.758346    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:58:43.758800 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.758740    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.759019 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.758995    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63079b0-5263-47db-9bb0-41eb3d415ca0-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.759153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.759027    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.759153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.759044    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.759153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.759060    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.759153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.759117    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.760247 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.760224    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.760448 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.760419    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:58:43.760565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.760516    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.760818 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.760741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt" pod "c63079b0-5263-47db-9bb0-41eb3d415ca0" (UID: "c63079b0-5263-47db-9bb0-41eb3d415ca0"). InnerVolumeSpecName "kube-api-access-hblpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.846204 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.845232    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 00:58:43.849562 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:58:43.849524    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50e39bf_f364_4fc1_b958_9a5a251d0804.slice/crio-96fb6079a204850dbbe8e160735cfc010930923584a30c8e89f3ee322ab34eb1 WatchSource:0}: Error finding container 96fb6079a204850dbbe8e160735cfc010930923584a30c8e89f3ee322ab34eb1: Status 404 returned error can't find the container with id 96fb6079a204850dbbe8e160735cfc010930923584a30c8e89f3ee322ab34eb1
Nov 01 00:58:43.859565 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859535    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.859692 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859633    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.859758 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859715    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.859811 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859752    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.859863 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859826    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.860305 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.859952    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.860305 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860042    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.860305 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860071    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.860305 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860177    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\") pod \"61cf6c5e-cae8-4e69-b061-2244b8732a49\" (UID: \"61cf6c5e-cae8-4e69-b061-2244b8732a49\") "
Nov 01 00:58:43.860611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860464    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.860611 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860534    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 00:58:43.861028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860852    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.861028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860929    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861028 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.860962    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861081    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf6c5e-cae8-4e69-b061-2244b8732a49-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861097    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63079b0-5263-47db-9bb0-41eb3d415ca0-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861111    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblpt\" (UniqueName: \"kubernetes.io/projected/c63079b0-5263-47db-9bb0-41eb3d415ca0-kube-api-access-hblpt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861127    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63079b0-5263-47db-9bb0-41eb3d415ca0-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861171 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861160    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.861375 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861350    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.861422 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.861380    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.863253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.863189    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 00:58:43.863253 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.863225    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "kube-api-access-78xqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.863560 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.863541    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 00:58:43.863755 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.863734    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy" pod "61cf6c5e-cae8-4e69-b061-2244b8732a49" (UID: "61cf6c5e-cae8-4e69-b061-2244b8732a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 00:58:43.914631 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.914599    4493 generic.go:356] "Generic (PLEG): container finished" podID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerID="f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576" exitCode=0
Nov 01 00:58:43.914827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.914658    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"
Nov 01 00:58:43.914827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.914681    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" event={"ID":"e26aaa66-5506-4b70-b8ad-b2e77d633aeb","Type":"ContainerDied","Data":"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"}
Nov 01 00:58:43.914827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.914728    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k" event={"ID":"e26aaa66-5506-4b70-b8ad-b2e77d633aeb","Type":"ContainerDied","Data":"7492eedcb4b303a0f8cd65a845d3b607c670d2ccd9872e92f1c42243f6a6db24"}
Nov 01 00:58:43.914827 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.914755    4493 scope.go:122] "RemoveContainer" containerID="f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"
Nov 01 00:58:43.935040 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.923197    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" event={"ID":"f50e39bf-f364-4fc1-b958-9a5a251d0804","Type":"ContainerStarted","Data":"96fb6079a204850dbbe8e160735cfc010930923584a30c8e89f3ee322ab34eb1"}
Nov 01 00:58:43.951573 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.951536    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf" event={"ID":"c63079b0-5263-47db-9bb0-41eb3d415ca0","Type":"ContainerDied","Data":"de86be7e8b95a897e7193e13df56c7acbd7fcfc9bba0c81b4a9d4e5875e10429"}
Nov 01 00:58:43.957569 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.957533    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp" event={"ID":"61cf6c5e-cae8-4e69-b061-2244b8732a49","Type":"ContainerDied","Data":"6ecf8c60d05edd10b3b822787238d85deeb80fc83a2fea494541fd6c84eaa97a"}
Nov 01 00:58:43.957746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.953870    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"
Nov 01 00:58:43.957746 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.951879    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"
Nov 01 00:58:43.965592 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.965563    4493 scope.go:122] "RemoveContainer" containerID="f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967606    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967638    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:43.967648    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576\": container with ID starting with f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576 not found: ID does not exist" containerID="f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967690    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576"} err="failed to get container status \"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576\": rpc error: code = NotFound desc = could not find container \"f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576\": container with ID starting with f18382b96afc92a04ead2398d57dfc1e7f9ef1ac227de9a9d285e08fb4b53576 not found: ID does not exist"
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967717    4493 scope.go:122] "RemoveContainer" containerID="170702f1870773dc46f820970c78cd0791c254d6cbc2b84eb906946ab06be7b3"
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967666    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967816    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967839    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967856    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xqq\" (UniqueName: \"kubernetes.io/projected/61cf6c5e-cae8-4e69-b061-2244b8732a49-kube-api-access-78xqq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.968401 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.967873    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf6c5e-cae8-4e69-b061-2244b8732a49-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 00:58:43.985952 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:43.985931    4493 scope.go:122] "RemoveContainer" containerID="0d763b801e01851a07954de5ac994bb3df8904b46ffcc46da3c93aacf2f3ea4b"
Nov 01 00:58:44.153448 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.153410    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:58:44.160932 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.160904    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-9t48k"]
Nov 01 00:58:44.323807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.323773    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:58:44.327829 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.327795    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-96hxp"]
Nov 01 00:58:44.439830 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.439787    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:58:44.443361 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.443326    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-vxqbf"]
Nov 01 00:58:44.504468 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.504434    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" path="/var/lib/kubelet/pods/61cf6c5e-cae8-4e69-b061-2244b8732a49/volumes"
Nov 01 00:58:44.504963 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.504949    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf82d413-4b6c-4871-8dc6-dfef0a86abfb" path="/var/lib/kubelet/pods/bf82d413-4b6c-4871-8dc6-dfef0a86abfb/volumes"
Nov 01 00:58:44.505406 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.505393    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" path="/var/lib/kubelet/pods/c63079b0-5263-47db-9bb0-41eb3d415ca0/volumes"
Nov 01 00:58:44.506162 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.506120    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" path="/var/lib/kubelet/pods/e26aaa66-5506-4b70-b8ad-b2e77d633aeb/volumes"
Nov 01 00:58:44.971768 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.971731    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" event={"ID":"f50e39bf-f364-4fc1-b958-9a5a251d0804","Type":"ContainerStarted","Data":"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"}
Nov 01 00:58:44.971960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:44.971791    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:44.994226 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:44.994111    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:58:45.095619 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:45.095551    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" podStartSLOduration=2.218324057 podStartE2EDuration="3.09553143s" totalImagesPullingTime="877.207373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:45.089557524 +0000 UTC m=+2114.767144492" watchObservedRunningTime="2026-11-01 00:58:45.09553143 +0000 UTC m=+2114.773118376"
Nov 01 00:58:49.016460 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.016426    4493 generic.go:356] "Generic (PLEG): container finished" podID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerID="29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e" exitCode=0
Nov 01 00:58:49.016902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.016498    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerDied","Data":"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"}
Nov 01 00:58:49.016977 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.016963    4493 scope.go:122] "RemoveContainer" containerID="29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"
Nov 01 00:58:49.018678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.018651    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:58:49.018751 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.018726    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:58:49.018786 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:49.018760    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:58:50.051000 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:50.050966    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerStarted","Data":"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"}
Nov 01 00:58:50.140159 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:50.138715    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:58:50.140359 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:50.140304    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:50.140410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:50.140356    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:50.989382 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:50.989353    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 00:58:51.135599 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:51.135563    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:51.136187 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:51.136153    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:52.138161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.135571    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:52.138161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.135637    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:52.842314 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.842265    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 00:58:52.843378 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843355    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy"
Nov 01 00:58:52.843435 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843382    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy"
Nov 01 00:58:52.843476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843445    4493 cpu_manager.go:414] "containerMap: removing container" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy"
Nov 01 00:58:52.843476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843454    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy"
Nov 01 00:58:52.843476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843464    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy"
Nov 01 00:58:52.843476 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843472    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy"
Nov 01 00:58:52.843647 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843635    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63079b0-5263-47db-9bb0-41eb3d415ca0" containerName="istio-proxy"
Nov 01 00:58:52.843681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843662    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cf6c5e-cae8-4e69-b061-2244b8732a49" containerName="istio-proxy"
Nov 01 00:58:52.843681 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.843672    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26aaa66-5506-4b70-b8ad-b2e77d633aeb" containerName="istio-proxy"
Nov 01 00:58:52.848073 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.848055    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:52.901972 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.901943    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 00:58:52.952880 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.952845    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-ba5b10-serving-cert\""
Nov 01 00:58:52.953119 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.952928    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwfx\" (UniqueName: \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:52.953437 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.953420    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:52.953538 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.953527    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:52.953616 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:52.953607    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.055086 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.055050    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.055281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.055095    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.055281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.055122    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.055281 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.055272    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwfx\" (UniqueName: \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.056165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.056111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.056272 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:53.056207    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-serving-cert: secret "maas-api-e2e-shared-a-w6-ba5b10-serving-cert" not found
Nov 01 00:58:53.056745 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:58:53.056724    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls podName:e878bb21-6c71-4b49-9b17-8c3297deca21 nodeName:}" failed. No retries permitted until 2026-11-01 00:58:53.556697838 +0000 UTC m=+2123.234284775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls") pod "maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" (UID: "e878bb21-6c71-4b49-9b17-8c3297deca21") : secret "maas-api-e2e-shared-a-w6-ba5b10-serving-cert" not found
Nov 01 00:58:53.061943 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.061907    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.148289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.147599    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:53.148289 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.147676    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:53.177396 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.172940    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwfx\" (UniqueName: \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.575912 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.575867    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.578595 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.578567    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:53.762254 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:53.762217    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:54.133733 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:54.133637    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:54.133984 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:54.133957    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:54.166721 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:54.166693    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 00:58:54.167438 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:58:54.167414    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode878bb21_6c71_4b49_9b17_8c3297deca21.slice/crio-905dce0e3d695e4a47e6be37b4a609aecb27bf7a8d847f6ddbae1aab349a9fa7 WatchSource:0}: Error finding container 905dce0e3d695e4a47e6be37b4a609aecb27bf7a8d847f6ddbae1aab349a9fa7: Status 404 returned error can't find the container with id 905dce0e3d695e4a47e6be37b4a609aecb27bf7a8d847f6ddbae1aab349a9fa7
Nov 01 00:58:55.119767 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:55.119720    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" event={"ID":"e878bb21-6c71-4b49-9b17-8c3297deca21","Type":"ContainerStarted","Data":"905dce0e3d695e4a47e6be37b4a609aecb27bf7a8d847f6ddbae1aab349a9fa7"}
Nov 01 00:58:55.139287 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:55.139250    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 01 00:58:55.140263 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:55.140230    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 01 00:58:56.161161 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.160865    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" event={"ID":"e878bb21-6c71-4b49-9b17-8c3297deca21","Type":"ContainerStarted","Data":"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"}
Nov 01 00:58:56.172960 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.172925    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:58:56.212164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.208748    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:58:56.212164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.210378    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:58:56.212164 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.212078    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 00:58:56.298727 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:58:56.298656    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" podStartSLOduration=3.310102634 podStartE2EDuration="4.298637251s" totalImagesPullingTime="988.534617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:56.297642647 +0000 UTC m=+2125.975229592" watchObservedRunningTime="2026-11-01 00:58:56.298637251 +0000 UTC m=+2125.976224191"
Nov 01 00:59:03.180208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:03.180178    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 00:59:05.036666 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.036622    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 00:59:05.041577 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.041552    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.103552 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.103518    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b112c158-openshift-default-dockercfg-jnjjd\""
Nov 01 00:59:05.126862 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.126822    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.126989    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127094    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127169 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127160    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127258 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127204    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127308 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127284    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127378 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127360    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127434 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127414    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.127528 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.127512    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vs4g\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.149116 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.149082    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 00:59:05.229340 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229303    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vs4g\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229367    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229450    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229531 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229503    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229537    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229557    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229629    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229678 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229654    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.229884 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.229696    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.230022 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.230001    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.230153 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.230105    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.230208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.230181    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.230270 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.230226    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.230330 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.230265    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.232113 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.232091    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.232368 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.232349    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.290626 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.290554    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.290871 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.290845    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vs4g\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.357311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.357275    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:05.796040 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.796002    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 00:59:05.796461 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:59:05.796437    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8171f6_7f06_4ce7_9d0b_171ab8721445.slice/crio-394cb115cc833013b99718e810a45ac2338514b3ac442049ba21c37f9d2a9fa3 WatchSource:0}: Error finding container 394cb115cc833013b99718e810a45ac2338514b3ac442049ba21c37f9d2a9fa3: Status 404 returned error can't find the container with id 394cb115cc833013b99718e810a45ac2338514b3ac442049ba21c37f9d2a9fa3
Nov 01 00:59:05.799311 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.799279    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:59:05.799421 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.799373    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:59:05.799485 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:05.799420    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 00:59:06.237242 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.237198    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" event={"ID":"9b8171f6-7f06-4ce7-9d0b-171ab8721445","Type":"ContainerStarted","Data":"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"}
Nov 01 00:59:06.237779 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.237275    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" event={"ID":"9b8171f6-7f06-4ce7-9d0b-171ab8721445","Type":"ContainerStarted","Data":"394cb115cc833013b99718e810a45ac2338514b3ac442049ba21c37f9d2a9fa3"}
Nov 01 00:59:06.357726 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.357690    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:06.361411 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.361375    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:06.361625 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.361440    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:06.363732 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:06.363687    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podStartSLOduration=2.363672277 podStartE2EDuration="2.363672277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:06.359274537 +0000 UTC m=+2136.036861484" watchObservedRunningTime="2026-11-01 00:59:06.363672277 +0000 UTC m=+2136.041259221"
Nov 01 00:59:07.358089 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:07.358046    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:07.358542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:07.358105    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:08.366157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:08.358463    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:08.366157 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:08.358526    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:09.358692 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.358652    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:09.358907 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.358712    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:09.444690 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.444631    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 00:59:09.449564 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.449543    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.525008 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.524964    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 00:59:09.551280 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.551239    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2r8z\" (UniqueName: \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.551957 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.551929    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.603165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.600219    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e296f9\""
Nov 01 00:59:09.603165 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.600341    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e296f9-dockercfg-7wqzs\""
Nov 01 00:59:09.667208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.658183    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.667208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.658393    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2r8z\" (UniqueName: \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.667208 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.659826    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.905575 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.905533    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 00:59:09.908879 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.908851    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2r8z\" (UniqueName: \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\") pod \"payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:09.928922 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.915578    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 00:59:09.928922 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:09.915689    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.013421 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.013390    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.013650 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.013428    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5w6l\" (UniqueName: \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.065387 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.065351    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:10.119170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.115697    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.119170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.115755    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5w6l\" (UniqueName: \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.119170 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.117237    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.177638 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.177601    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5w6l\" (UniqueName: \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\") pod \"payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.259514 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.259473    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:10.358391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.357889    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:10.358391 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.357961    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:10.473542 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.473512    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 00:59:10.474084 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:59:10.474057    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19909aa7_543e_480a_acd2_84655e8abb4d.slice/crio-9ee7637a154ce91893e7e2e5bbf739a073c4344996af3b72ac6ed0841c9688d7 WatchSource:0}: Error finding container 9ee7637a154ce91893e7e2e5bbf739a073c4344996af3b72ac6ed0841c9688d7: Status 404 returned error can't find the container with id 9ee7637a154ce91893e7e2e5bbf739a073c4344996af3b72ac6ed0841c9688d7
Nov 01 00:59:10.663503 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:10.663478    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 00:59:10.664861 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:59:10.664832    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3100ef_68f3_4199_9aaf_e4ea15262a55.slice/crio-a78ab8c38c1c2fadb1bb93f8784b224efb0aeff5abde1a89c9f376b4e7c683b9 WatchSource:0}: Error finding container a78ab8c38c1c2fadb1bb93f8784b224efb0aeff5abde1a89c9f376b4e7c683b9: Status 404 returned error can't find the container with id a78ab8c38c1c2fadb1bb93f8784b224efb0aeff5abde1a89c9f376b4e7c683b9
Nov 01 00:59:11.299160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.298363    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" event={"ID":"19909aa7-543e-480a-acd2-84655e8abb4d","Type":"ContainerStarted","Data":"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"}
Nov 01 00:59:11.299160 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.298428    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" event={"ID":"19909aa7-543e-480a-acd2-84655e8abb4d","Type":"ContainerStarted","Data":"9ee7637a154ce91893e7e2e5bbf739a073c4344996af3b72ac6ed0841c9688d7"}
Nov 01 00:59:11.299594 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.299569    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:11.303752 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.303722    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" event={"ID":"6f3100ef-68f3-4199-9aaf-e4ea15262a55","Type":"ContainerStarted","Data":"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"}
Nov 01 00:59:11.303851 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.303769    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" event={"ID":"6f3100ef-68f3-4199-9aaf-e4ea15262a55","Type":"ContainerStarted","Data":"a78ab8c38c1c2fadb1bb93f8784b224efb0aeff5abde1a89c9f376b4e7c683b9"}
Nov 01 00:59:11.304212 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.304197    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:11.357882 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.357850    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:11.358047 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.357917    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:11.442532 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.442482    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" podStartSLOduration=2.442463922 podStartE2EDuration="2.442463922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:11.414840941 +0000 UTC m=+2141.092427888" watchObservedRunningTime="2026-11-01 00:59:11.442463922 +0000 UTC m=+2141.120050866"
Nov 01 00:59:11.539158 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:11.535631    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" podStartSLOduration=2.535609487 podStartE2EDuration="2.535609487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:11.533685496 +0000 UTC m=+2141.211272441" watchObservedRunningTime="2026-11-01 00:59:11.535609487 +0000 UTC m=+2141.213196433"
Nov 01 00:59:12.358044 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:12.358010    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 00:59:12.358256 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:12.358069    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 00:59:13.363053 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:13.363022    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:13.363524 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:13.363482    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:13.364366 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:13.364347    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 00:59:14.993864 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:59:14.993822    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:59:17.972166 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:17.959107    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 00:59:18.102635 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.102604    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.144854 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.144816    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d652r\" (UniqueName: \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.145084 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.145016    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.152779 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.152746    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 00:59:18.162517 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.162475    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-bc241b\""
Nov 01 00:59:18.212035 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.212001    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-bc241b-dockercfg-vfcpp\""
Nov 01 00:59:18.246516 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.246418    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d652r\" (UniqueName: \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.246704 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.246552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.247410 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.247383    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.307477 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.307440    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d652r\" (UniqueName: \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\") pod \"payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.460050 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.460017    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:18.841807 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:18.841777    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 00:59:18.842114 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:59:18.842092    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ef3bd9_8823_4b6d_a1ab_6aa508ea32d0.slice/crio-98a438a650f148b5a4406de676267f7a37797220ed78cdb798a4cee2dd114e14 WatchSource:0}: Error finding container 98a438a650f148b5a4406de676267f7a37797220ed78cdb798a4cee2dd114e14: Status 404 returned error can't find the container with id 98a438a650f148b5a4406de676267f7a37797220ed78cdb798a4cee2dd114e14
Nov 01 00:59:19.479301 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:19.479267    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" event={"ID":"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0","Type":"ContainerStarted","Data":"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"}
Nov 01 00:59:19.479717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:19.479318    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" event={"ID":"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0","Type":"ContainerStarted","Data":"98a438a650f148b5a4406de676267f7a37797220ed78cdb798a4cee2dd114e14"}
Nov 01 00:59:19.479717 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:19.479393    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:19.595666 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:19.595617    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" podStartSLOduration=2.595603803 podStartE2EDuration="2.595603803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:19.594456009 +0000 UTC m=+2149.272042953" watchObservedRunningTime="2026-11-01 00:59:19.595603803 +0000 UTC m=+2149.273190812"
Nov 01 00:59:22.314201 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:22.314170    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 00:59:23.320687 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:23.320660    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 00:59:23.586413 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:23.586313    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-dpkl7" podUID="da6e216e-f24a-4c74-97c0-7de00ca15f30" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.18:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 00:59:25.137942 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.137908    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 00:59:25.142385 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.142364    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.247302 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.247265    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f97e89-serving-cert\""
Nov 01 00:59:25.252099 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.252073    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 00:59:25.254763 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.254733    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.254873 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.254817    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvdd\" (UniqueName: \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.254968 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.254951    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.255034 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.255015    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.356288 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.356244    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.356511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.356315    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvdd\" (UniqueName: \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.356511 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:59:25.356407    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-serving-cert: secret "maas-api-e2e-api-a-f97e89-serving-cert" not found
Nov 01 00:59:25.356511 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:59:25.356479    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls podName:b6e8b9f2-3483-4e53-b08a-f4b94c908dfe nodeName:}" failed. No retries permitted until 2026-11-01 00:59:25.85646237 +0000 UTC m=+2155.534049306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls") pod "maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" (UID: "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe") : secret "maas-api-e2e-api-a-f97e89-serving-cert" not found
Nov 01 00:59:25.356511 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.356414    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.356749 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.356567    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.357194 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.357168    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.359178 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.359150    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.429322 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.429235    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvdd\" (UniqueName: \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.867492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.867458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:25.870068 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:25.870046    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") pod \"maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:26.056492 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:26.056456    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:26.446990 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:26.446959    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 00:59:26.447408 ip-10-0-141-75 kubenswrapper[4493]: W1101 00:59:26.447333    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e8b9f2_3483_4e53_b08a_f4b94c908dfe.slice/crio-f20cb95b49065db76efe2fbacf6228fdcb9dc462a027dbae1e971d1fb7636e8f WatchSource:0}: Error finding container f20cb95b49065db76efe2fbacf6228fdcb9dc462a027dbae1e971d1fb7636e8f: Status 404 returned error can't find the container with id f20cb95b49065db76efe2fbacf6228fdcb9dc462a027dbae1e971d1fb7636e8f
Nov 01 00:59:26.533167 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:26.533122    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" event={"ID":"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe","Type":"ContainerStarted","Data":"f20cb95b49065db76efe2fbacf6228fdcb9dc462a027dbae1e971d1fb7636e8f"}
Nov 01 00:59:27.542371 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:27.542330    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" event={"ID":"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe","Type":"ContainerStarted","Data":"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"}
Nov 01 00:59:27.542750 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:27.542431    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:27.685902 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:27.685792    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" podStartSLOduration=1.791908332 podStartE2EDuration="2.685776021s" totalImagesPullingTime="893.867689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:27.683694046 +0000 UTC m=+2157.361281015" watchObservedRunningTime="2026-11-01 00:59:27.685776021 +0000 UTC m=+2157.363362966"
Nov 01 00:59:30.491213 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:30.491180    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 00:59:33.556074 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:33.556044    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 00:59:44.991930 ip-10-0-141-75 kubenswrapper[4493]: E1101 00:59:44.991888    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 00:59:54.789689 ip-10-0-141-75 kubenswrapper[4493]: I1101 00:59:54.789644    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:00:00.226666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.226625    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:00.232351 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.232332    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:00.232458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.232447    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.236661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.236640    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.360596 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.360564    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:00:00.382938 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.382902    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.382938 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.382944    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprjk\" (UniqueName: \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.383366 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.382970    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.383366 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.383100    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnkz\" (UniqueName: \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.440933 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.440891    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:00.482918 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.482827    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:00.485414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.485387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.485666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.485647    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprjk\" (UniqueName: \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.485837 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.485814    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.485969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.485905    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnkz\" (UniqueName: \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.486582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.486559    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.488512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.488487    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.547953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.547923    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnkz\" (UniqueName: \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.548115 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.548026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprjk\" (UniqueName: \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\") pod \"maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.564302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.564269    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:00.571928 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.571908    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:00.954966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:00.954885    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:00.991850 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:00:00.991823    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74a428a_e76d_40bf_b0c7_1f5d8b9ffa90.slice/crio-7cb06902ece601cf72bb1e06571fac50f22478e34ec6b5c7caf0cbc249e67952 WatchSource:0}: Error finding container 7cb06902ece601cf72bb1e06571fac50f22478e34ec6b5c7caf0cbc249e67952: Status 404 returned error can't find the container with id 7cb06902ece601cf72bb1e06571fac50f22478e34ec6b5c7caf0cbc249e67952
Nov 01 01:00:01.009313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.009285    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:01.009549 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:00:01.009526    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab117d7_b435_4947_b1a1_0def934933a0.slice/crio-f3b6eaaba6c1a6c3978f32210bffaca2996fcfc1df0e2b8bf418c30bfc1d81e7 WatchSource:0}: Error finding container f3b6eaaba6c1a6c3978f32210bffaca2996fcfc1df0e2b8bf418c30bfc1d81e7: Status 404 returned error can't find the container with id f3b6eaaba6c1a6c3978f32210bffaca2996fcfc1df0e2b8bf418c30bfc1d81e7
Nov 01 01:00:01.807064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.807033    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerStarted","Data":"98712376b00ee9ec27605e697f5d3e0341aa873486360bdabedc806700c4640e"}
Nov 01 01:00:01.807569 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.807083    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerStarted","Data":"7cb06902ece601cf72bb1e06571fac50f22478e34ec6b5c7caf0cbc249e67952"}
Nov 01 01:00:01.808756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.808733    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerStarted","Data":"e89360e48fc1d4e3da9455eb300671833a3daccb2d02dbac3835a8d056e8f9b9"}
Nov 01 01:00:01.808871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.808775    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerStarted","Data":"f3b6eaaba6c1a6c3978f32210bffaca2996fcfc1df0e2b8bf418c30bfc1d81e7"}
Nov 01 01:00:01.923825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:01.923543    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" podStartSLOduration=1.923522981 podStartE2EDuration="1.923522981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:01.922647629 +0000 UTC m=+2191.600234575" watchObservedRunningTime="2026-11-01 01:00:01.923522981 +0000 UTC m=+2191.601109928"
Nov 01 01:00:02.043900 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.043851    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" podStartSLOduration=2.04383633 podStartE2EDuration="2.04383633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.040578982 +0000 UTC m=+2191.718165929" watchObservedRunningTime="2026-11-01 01:00:02.04383633 +0000 UTC m=+2191.721423274"
Nov 01 01:00:02.820920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.820885    4493 generic.go:356] "Generic (PLEG): container finished" podID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerID="98712376b00ee9ec27605e697f5d3e0341aa873486360bdabedc806700c4640e" exitCode=60
Nov 01 01:00:02.821436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.820977    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerDied","Data":"98712376b00ee9ec27605e697f5d3e0341aa873486360bdabedc806700c4640e"}
Nov 01 01:00:02.821436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.821327    4493 scope.go:122] "RemoveContainer" containerID="98712376b00ee9ec27605e697f5d3e0341aa873486360bdabedc806700c4640e"
Nov 01 01:00:02.822831 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.822809    4493 generic.go:356] "Generic (PLEG): container finished" podID="bab117d7-b435-4947-b1a1-0def934933a0" containerID="e89360e48fc1d4e3da9455eb300671833a3daccb2d02dbac3835a8d056e8f9b9" exitCode=60
Nov 01 01:00:02.822955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.822926    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerDied","Data":"e89360e48fc1d4e3da9455eb300671833a3daccb2d02dbac3835a8d056e8f9b9"}
Nov 01 01:00:02.823112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:02.823099    4493 scope.go:122] "RemoveContainer" containerID="e89360e48fc1d4e3da9455eb300671833a3daccb2d02dbac3835a8d056e8f9b9"
Nov 01 01:00:03.843710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.843678    4493 generic.go:356] "Generic (PLEG): container finished" podID="bab117d7-b435-4947-b1a1-0def934933a0" containerID="ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890" exitCode=60
Nov 01 01:00:03.844219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.843757    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerDied","Data":"ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890"}
Nov 01 01:00:03.844219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.843846    4493 scope.go:122] "RemoveContainer" containerID="e89360e48fc1d4e3da9455eb300671833a3daccb2d02dbac3835a8d056e8f9b9"
Nov 01 01:00:03.844219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.844189    4493 scope.go:122] "RemoveContainer" containerID="ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890"
Nov 01 01:00:03.844574 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:03.844551    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8_odh-ai-gateway-infra(bab117d7-b435-4947-b1a1-0def934933a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" podUID="bab117d7-b435-4947-b1a1-0def934933a0"
Nov 01 01:00:03.846111 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.846090    4493 generic.go:356] "Generic (PLEG): container finished" podID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerID="943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34" exitCode=60
Nov 01 01:00:03.846237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.846179    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerDied","Data":"943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34"}
Nov 01 01:00:03.846473 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.846457    4493 scope.go:122] "RemoveContainer" containerID="943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34"
Nov 01 01:00:03.846774 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:03.846755    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj_odh-ai-gateway-infra(a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90"
Nov 01 01:00:03.856980 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:03.856961    4493 scope.go:122] "RemoveContainer" containerID="98712376b00ee9ec27605e697f5d3e0341aa873486360bdabedc806700c4640e"
Nov 01 01:00:04.856479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:04.856451    4493 scope.go:122] "RemoveContainer" containerID="943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34"
Nov 01 01:00:04.856934 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:04.856854    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj_odh-ai-gateway-infra(a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90"
Nov 01 01:00:04.858210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:04.858188    4493 scope.go:122] "RemoveContainer" containerID="ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890"
Nov 01 01:00:04.858528 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:04.858507    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8_odh-ai-gateway-infra(bab117d7-b435-4947-b1a1-0def934933a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" podUID="bab117d7-b435-4947-b1a1-0def934933a0"
Nov 01 01:00:14.991589 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:14.991510    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:00:16.495890 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:16.495861    4493 scope.go:122] "RemoveContainer" containerID="943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34"
Nov 01 01:00:16.967834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:16.967800    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerStarted","Data":"269408c7460d42826ff2f5d548c9255029839a6507bb8d0b02b1abe75f5d6273"}
Nov 01 01:00:17.495819 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.495791    4493 scope.go:122] "RemoveContainer" containerID="ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890"
Nov 01 01:00:17.976245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.976193    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerStarted","Data":"e2d4fbf215aedaeba990b499881a9a42efba038ebd195cc0d7bef70f653cd284"}
Nov 01 01:00:17.978439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.978412    4493 generic.go:356] "Generic (PLEG): container finished" podID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerID="269408c7460d42826ff2f5d548c9255029839a6507bb8d0b02b1abe75f5d6273" exitCode=60
Nov 01 01:00:17.978590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.978480    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerDied","Data":"269408c7460d42826ff2f5d548c9255029839a6507bb8d0b02b1abe75f5d6273"}
Nov 01 01:00:17.978590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.978535    4493 scope.go:122] "RemoveContainer" containerID="943f76db10511573dd4a0d77ff7c53c73221fa0f365e941b46b8647a5710dd34"
Nov 01 01:00:17.978891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:17.978875    4493 scope.go:122] "RemoveContainer" containerID="269408c7460d42826ff2f5d548c9255029839a6507bb8d0b02b1abe75f5d6273"
Nov 01 01:00:17.979282 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:17.979257    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj_odh-ai-gateway-infra(a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90"
Nov 01 01:00:18.100365 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:18.100338    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:18.987323 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:18.987293    4493 generic.go:356] "Generic (PLEG): container finished" podID="bab117d7-b435-4947-b1a1-0def934933a0" containerID="e2d4fbf215aedaeba990b499881a9a42efba038ebd195cc0d7bef70f653cd284" exitCode=60
Nov 01 01:00:18.987690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:18.987363    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerDied","Data":"e2d4fbf215aedaeba990b499881a9a42efba038ebd195cc0d7bef70f653cd284"}
Nov 01 01:00:18.987690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:18.987427    4493 scope.go:122] "RemoveContainer" containerID="ebd523ddbbbba25b84c5465ad94e44e3469734a4b3f6a77cdea0855648fb0890"
Nov 01 01:00:18.987889 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:18.987875    4493 scope.go:122] "RemoveContainer" containerID="e2d4fbf215aedaeba990b499881a9a42efba038ebd195cc0d7bef70f653cd284"
Nov 01 01:00:18.988266 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:18.988241    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8_odh-ai-gateway-infra(bab117d7-b435-4947-b1a1-0def934933a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" podUID="bab117d7-b435-4947-b1a1-0def934933a0"
Nov 01 01:00:19.122406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.122374    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:19.153570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.153543    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:19.349522 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.349430    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\") pod \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") "
Nov 01 01:00:19.361386 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.354341    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca" pod "a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" (UID: "a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:00:19.361386 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.360968    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\" (UniqueName: \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\") pod \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\" (UID: \"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90\") "
Nov 01 01:00:19.362114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.362085    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:00:19.364325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.364252    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz" pod "a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" (UID: "a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90"). InnerVolumeSpecName "kube-api-access-mtnkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:00:19.463121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.463088    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtnkz\" (UniqueName: \"kubernetes.io/projected/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90-kube-api-access-mtnkz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:00:19.998363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.998330    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"
Nov 01 01:00:19.998363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.998358    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj" event={"ID":"a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90","Type":"ContainerDied","Data":"7cb06902ece601cf72bb1e06571fac50f22478e34ec6b5c7caf0cbc249e67952"}
Nov 01 01:00:19.998934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:19.998420    4493 scope.go:122] "RemoveContainer" containerID="269408c7460d42826ff2f5d548c9255029839a6507bb8d0b02b1abe75f5d6273"
Nov 01 01:00:20.163514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.163485    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:20.277048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.276965    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\" (UniqueName: \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\") pod \"bab117d7-b435-4947-b1a1-0def934933a0\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") "
Nov 01 01:00:20.277206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.277109    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\") pod \"bab117d7-b435-4947-b1a1-0def934933a0\" (UID: \"bab117d7-b435-4947-b1a1-0def934933a0\") "
Nov 01 01:00:20.277524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.277497    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca" pod "bab117d7-b435-4947-b1a1-0def934933a0" (UID: "bab117d7-b435-4947-b1a1-0def934933a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:00:20.277839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.277824    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab117d7-b435-4947-b1a1-0def934933a0-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:00:20.279538 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.279512    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk" pod "bab117d7-b435-4947-b1a1-0def934933a0" (UID: "bab117d7-b435-4947-b1a1-0def934933a0"). InnerVolumeSpecName "kube-api-access-sprjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:00:20.286405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.286378    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:20.296350 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.296329    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891580-b8jhj"]
Nov 01 01:00:20.379769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.379731    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprjk\" (UniqueName: \"kubernetes.io/projected/bab117d7-b435-4947-b1a1-0def934933a0-kube-api-access-sprjk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:00:20.505793 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.505762    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" path="/var/lib/kubelet/pods/a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90/volumes"
Nov 01 01:00:20.933413 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.933110    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934302    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934323    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934342    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934350    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934377    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934384    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934397    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934406    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934454    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934460    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934616    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934628    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934641    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.934656    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:20.942212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.939225    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:20.991613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.991579    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:00:20.993389 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.993359    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e296f9-serving-cert\""
Nov 01 01:00:20.997382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.997350    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:20.997622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.997602    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:20.997719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.997701    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5p6t\" (UniqueName: \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:20.997776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:20.997746    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.014108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.014076    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"
Nov 01 01:00:21.014590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.014115    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8" event={"ID":"bab117d7-b435-4947-b1a1-0def934933a0","Type":"ContainerDied","Data":"f3b6eaaba6c1a6c3978f32210bffaca2996fcfc1df0e2b8bf418c30bfc1d81e7"}
Nov 01 01:00:21.014590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.014197    4493 scope.go:122] "RemoveContainer" containerID="e2d4fbf215aedaeba990b499881a9a42efba038ebd195cc0d7bef70f653cd284"
Nov 01 01:00:21.099294 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.099251    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.099512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.099335    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.099512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.099409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5p6t\" (UniqueName: \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.099512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.099445    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.099512 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:21.099463    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-serving-cert: secret "maas-api-e2e-shared-b-w1-e296f9-serving-cert" not found
Nov 01 01:00:21.099737 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:21.099531    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls podName:42933fe6-2288-4863-8602-c47e638d1619 nodeName:}" failed. No retries permitted until 2026-11-01 01:00:21.599511161 +0000 UTC m=+2211.277098085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls") pod "maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" (UID: "42933fe6-2288-4863-8602-c47e638d1619") : secret "maas-api-e2e-shared-b-w1-e296f9-serving-cert" not found
Nov 01 01:00:21.100105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.100083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.101906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.101878    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.166685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.166648    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5p6t\" (UniqueName: \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.179211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.179176    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:21.190884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.190807    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b112c158-29891580-sj7d8"]
Nov 01 01:00:21.693650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.693610    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.696497 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.696462    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:21.870779 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:21.864482    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:22.506552 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:22.506516    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab117d7-b435-4947-b1a1-0def934933a0" path="/var/lib/kubelet/pods/bab117d7-b435-4947-b1a1-0def934933a0/volumes"
Nov 01 01:00:22.751292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:22.751262    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:00:22.755090 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:00:22.755057    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42933fe6_2288_4863_8602_c47e638d1619.slice/crio-e642878b54c5bd770511bbf7bc8741be8304940ea886c8f8deae360694f02d70 WatchSource:0}: Error finding container e642878b54c5bd770511bbf7bc8741be8304940ea886c8f8deae360694f02d70: Status 404 returned error can't find the container with id e642878b54c5bd770511bbf7bc8741be8304940ea886c8f8deae360694f02d70
Nov 01 01:00:23.365665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:23.365624    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" event={"ID":"42933fe6-2288-4863-8602-c47e638d1619","Type":"ContainerStarted","Data":"e642878b54c5bd770511bbf7bc8741be8304940ea886c8f8deae360694f02d70"}
Nov 01 01:00:24.374237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:24.374202    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" event={"ID":"42933fe6-2288-4863-8602-c47e638d1619","Type":"ContainerStarted","Data":"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"}
Nov 01 01:00:24.374634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:24.374358    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:24.505193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:24.505112    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" podStartSLOduration=3.581949916 podStartE2EDuration="4.505093819s" totalImagesPullingTime="923.143903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:24.50443113 +0000 UTC m=+2214.182018079" watchObservedRunningTime="2026-11-01 01:00:24.505093819 +0000 UTC m=+2214.182680764"
Nov 01 01:00:30.387237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:30.387202    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:00:37.033949 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.033917    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:00:37.034661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.034645    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:37.034707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.034662    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:37.034793 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.034784    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a428a-e76d-40bf-b0c7-1f5d8b9ffa90" containerName="cleanup"
Nov 01 01:00:37.034828 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.034797    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab117d7-b435-4947-b1a1-0def934933a0" containerName="cleanup"
Nov 01 01:00:37.039825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.039803    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.120467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.120429    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngdm\" (UniqueName: \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.120626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.120543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.120668 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.120650    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.120827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.120803    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.142316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.142284    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-bc241b-serving-cert\""
Nov 01 01:00:37.146721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.146697    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:00:37.222197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.222123    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.222381 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.222248    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngdm\" (UniqueName: \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.222381 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.222330    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.222508 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.222474    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.222508 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:37.222483    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-serving-cert: secret "maas-api-e2e-shared-b-w4-bc241b-serving-cert" not found
Nov 01 01:00:37.222636 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:37.222568    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls podName:811fabb5-5e40-4002-bed0-2fab51c7d065 nodeName:}" failed. No retries permitted until 2026-11-01 01:00:37.722546694 +0000 UTC m=+2227.400133635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls") pod "maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" (UID: "811fabb5-5e40-4002-bed0-2fab51c7d065") : secret "maas-api-e2e-shared-b-w4-bc241b-serving-cert" not found
Nov 01 01:00:37.223110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.223059    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.224935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.224912    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.298833 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.298754    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngdm\" (UniqueName: \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.737478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.737444    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.740096 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.740072    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:37.954372 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:37.954341    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:38.343516 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:38.343482    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:00:38.344604 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:00:38.344577    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811fabb5_5e40_4002_bed0_2fab51c7d065.slice/crio-155c4e8a4e6dc9b4b46ce3ef243f30924d79d17d04553d9592d6cd51aeef3a80 WatchSource:0}: Error finding container 155c4e8a4e6dc9b4b46ce3ef243f30924d79d17d04553d9592d6cd51aeef3a80: Status 404 returned error can't find the container with id 155c4e8a4e6dc9b4b46ce3ef243f30924d79d17d04553d9592d6cd51aeef3a80
Nov 01 01:00:38.543719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:38.543677    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" event={"ID":"811fabb5-5e40-4002-bed0-2fab51c7d065","Type":"ContainerStarted","Data":"155c4e8a4e6dc9b4b46ce3ef243f30924d79d17d04553d9592d6cd51aeef3a80"}
Nov 01 01:00:39.552869 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:39.552785    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" event={"ID":"811fabb5-5e40-4002-bed0-2fab51c7d065","Type":"ContainerStarted","Data":"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"}
Nov 01 01:00:39.553274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:39.552921    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:39.670706 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:39.670637    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" podStartSLOduration=2.748058446 podStartE2EDuration="3.670618963s" totalImagesPullingTime="922.560517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:39.67025746 +0000 UTC m=+2229.347844409" watchObservedRunningTime="2026-11-01 01:00:39.670618963 +0000 UTC m=+2229.348205910"
Nov 01 01:00:45.018857 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:00:45.018819    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:00:45.641382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:45.641350    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:00:45.794830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:00:45.794794    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:01:14.991678 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:01:14.991637    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:01:18.767355 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:18.767312    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 01:01:18.862185 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:18.862148    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 01:01:23.768400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:23.768355    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing" containerID="cri-o://7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7" gracePeriod=25
Nov 01 01:01:23.862780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:23.862737    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing" containerID="cri-o://37c8b91ef0af284fbeb359b88f345a2c5af4ec751f104229cfc3d0f5ef6a111c" gracePeriod=25
Nov 01 01:01:28.940682 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:28.940642    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:01:30.404311 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:30.404276    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:01:30.404719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:30.404280    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:01:40.403957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:40.403913    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:01:40.404457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:40.404343    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:01:43.117875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.117795    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:01:43.122182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.122163    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.174575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.174550    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0f9d81-serving-cert\""
Nov 01 01:01:43.228839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.228808    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:01:43.272565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.272528    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.272763 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.272652    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.272763 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.272741    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccj2\" (UniqueName: \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.272899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.272846    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.394535 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.394441    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.436567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.436529    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.436866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.436848    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccj2\" (UniqueName: \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.437033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.437018    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.437468 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:01:43.437422    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-serving-cert: secret "maas-api-e2e-inf-a-0f9d81-serving-cert" not found
Nov 01 01:01:43.437689 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:01:43.437677    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls podName:13079830-5111-4d67-873a-812bd9038a5e nodeName:}" failed. No retries permitted until 2026-11-01 01:01:43.937655112 +0000 UTC m=+2293.615242058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls") pod "maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" (UID: "13079830-5111-4d67-873a-812bd9038a5e") : secret "maas-api-e2e-inf-a-0f9d81-serving-cert" not found
Nov 01 01:01:43.438070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.430410    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.451187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.438979    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.499654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.499621    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccj2\" (UniqueName: \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.945117 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.945081    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:43.947693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:43.947669    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:44.037471 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.037432    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:44.133052 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.133026    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 01:01:44.188112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.187806    4493 generic.go:356] "Generic (PLEG): container finished" podID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerID="37c8b91ef0af284fbeb359b88f345a2c5af4ec751f104229cfc3d0f5ef6a111c" exitCode=0
Nov 01 01:01:44.188112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.188051    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" event={"ID":"722513e5-97ca-41c3-95df-6cddd56cb1d7","Type":"ContainerDied","Data":"37c8b91ef0af284fbeb359b88f345a2c5af4ec751f104229cfc3d0f5ef6a111c"}
Nov 01 01:01:44.190076 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.190050    4493 generic.go:356] "Generic (PLEG): container finished" podID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerID="7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7" exitCode=0
Nov 01 01:01:44.190524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.190193    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" event={"ID":"7ec90e9f-68ab-49cd-85de-4521d65aff26","Type":"ContainerDied","Data":"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"}
Nov 01 01:01:44.191281 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.190285    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"
Nov 01 01:01:44.191628 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.191607    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5" event={"ID":"7ec90e9f-68ab-49cd-85de-4521d65aff26","Type":"ContainerDied","Data":"dd751958efd64d4c4c511092d9c6ced4497f764e097574074696ca62b3c21058"}
Nov 01 01:01:44.191820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.191748    4493 scope.go:122] "RemoveContainer" containerID="7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"
Nov 01 01:01:44.206462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.206386    4493 scope.go:122] "RemoveContainer" containerID="7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"
Nov 01 01:01:44.207116 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:01:44.207055    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7\": container with ID starting with 7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7 not found: ID does not exist" containerID="7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"
Nov 01 01:01:44.207251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.207154    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7"} err="failed to get container status \"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7\": rpc error: code = NotFound desc = could not find container \"7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7\": container with ID starting with 7f1e411786d8bd1f23342d142cf0158789ac402296c58beeba78cf37782b05a7 not found: ID does not exist"
Nov 01 01:01:44.213272 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.213190    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 01:01:44.254297 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.254260    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\") pod \"7ec90e9f-68ab-49cd-85de-4521d65aff26\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") "
Nov 01 01:01:44.254455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.254437    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\" (UniqueName: \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\") pod \"722513e5-97ca-41c3-95df-6cddd56cb1d7\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") "
Nov 01 01:01:44.254536 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.254520    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\" (UniqueName: \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\") pod \"7ec90e9f-68ab-49cd-85de-4521d65aff26\" (UID: \"7ec90e9f-68ab-49cd-85de-4521d65aff26\") "
Nov 01 01:01:44.254721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.254686    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\") pod \"722513e5-97ca-41c3-95df-6cddd56cb1d7\" (UID: \"722513e5-97ca-41c3-95df-6cddd56cb1d7\") "
Nov 01 01:01:44.254848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.254712    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume" pod "7ec90e9f-68ab-49cd-85de-4521d65aff26" (UID: "7ec90e9f-68ab-49cd-85de-4521d65aff26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:01:44.255249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.255221    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume" pod "722513e5-97ca-41c3-95df-6cddd56cb1d7" (UID: "722513e5-97ca-41c3-95df-6cddd56cb1d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:01:44.255656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.255629    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722513e5-97ca-41c3-95df-6cddd56cb1d7-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:01:44.255750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.255659    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec90e9f-68ab-49cd-85de-4521d65aff26-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:01:44.257057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.257038    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q" pod "722513e5-97ca-41c3-95df-6cddd56cb1d7" (UID: "722513e5-97ca-41c3-95df-6cddd56cb1d7"). InnerVolumeSpecName "kube-api-access-h6x8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:01:44.257388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.257368    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk" pod "7ec90e9f-68ab-49cd-85de-4521d65aff26" (UID: "7ec90e9f-68ab-49cd-85de-4521d65aff26"). InnerVolumeSpecName "kube-api-access-rshrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:01:44.357289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.357219    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6x8q\" (UniqueName: \"kubernetes.io/projected/722513e5-97ca-41c3-95df-6cddd56cb1d7-kube-api-access-h6x8q\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:01:44.357289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.357246    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshrk\" (UniqueName: \"kubernetes.io/projected/7ec90e9f-68ab-49cd-85de-4521d65aff26-kube-api-access-rshrk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:01:44.518966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.518940    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:01:44.519778 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:01:44.519755    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13079830_5111_4d67_873a_812bd9038a5e.slice/crio-83f09676c3d45d1f32bd537150615a0f0696af8f414302d4c958b9e1aacb427b WatchSource:0}: Error finding container 83f09676c3d45d1f32bd537150615a0f0696af8f414302d4c958b9e1aacb427b: Status 404 returned error can't find the container with id 83f09676c3d45d1f32bd537150615a0f0696af8f414302d4c958b9e1aacb427b
Nov 01 01:01:44.626510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.626416    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 01:01:44.630686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:44.630662    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-hb4m5"]
Nov 01 01:01:44.991197 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:01:44.991160    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:01:45.198393 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.198353    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" event={"ID":"13079830-5111-4d67-873a-812bd9038a5e","Type":"ContainerStarted","Data":"83f09676c3d45d1f32bd537150615a0f0696af8f414302d4c958b9e1aacb427b"}
Nov 01 01:01:45.200875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.200845    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h" event={"ID":"722513e5-97ca-41c3-95df-6cddd56cb1d7","Type":"ContainerDied","Data":"ed853d4ab57a57f08c6105d03ea02c1fafb017a971ac71d52f60d1d345c04b3a"}
Nov 01 01:01:45.201021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.200897    4493 scope.go:122] "RemoveContainer" containerID="37c8b91ef0af284fbeb359b88f345a2c5af4ec751f104229cfc3d0f5ef6a111c"
Nov 01 01:01:45.201021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.200853    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"
Nov 01 01:01:45.373998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.373921    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 01:01:45.424057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:45.424025    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-zl92h"]
Nov 01 01:01:46.210077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:46.210043    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" event={"ID":"13079830-5111-4d67-873a-812bd9038a5e","Type":"ContainerStarted","Data":"752f036d7b5f7ee0592ac0d3f40e01c6669a190af7c37e03485dc5dd703e5b21"}
Nov 01 01:01:46.210699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:46.210197    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:46.354066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:46.347665    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" podStartSLOduration=2.392270149 podStartE2EDuration="3.347644212s" totalImagesPullingTime="955.374063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:46.340227533 +0000 UTC m=+2296.017814481" watchObservedRunningTime="2026-11-01 01:01:46.347644212 +0000 UTC m=+2296.025231158"
Nov 01 01:01:46.546642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:46.546604    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" path="/var/lib/kubelet/pods/722513e5-97ca-41c3-95df-6cddd56cb1d7/volumes"
Nov 01 01:01:46.550157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:46.547616    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" path="/var/lib/kubelet/pods/7ec90e9f-68ab-49cd-85de-4521d65aff26/volumes"
Nov 01 01:01:52.246567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:52.246535    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:01:56.511566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:56.511536    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 01:01:56.512000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:56.511783    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" containerID="cri-o://d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2" gracePeriod=30
Nov 01 01:01:57.302987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.302948    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:01:57.313950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.303216    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:01:57.754850 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.754817    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 01:01:57.755500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.755094    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy" containerID="cri-o://9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c" gracePeriod=30
Nov 01 01:01:57.758744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.758719    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 01:01:57.758929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.758905    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" containerID="cri-o://72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066" gracePeriod=30
Nov 01 01:01:57.805191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.805153    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 01:01:57.805424 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:57.805398    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" containerID="cri-o://b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01" gracePeriod=30
Nov 01 01:01:58.144164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:58.144044    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 01:01:58.144387 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:58.144342    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" containerID="cri-o://68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91" gracePeriod=30
Nov 01 01:01:58.148192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:58.148167    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 01:01:58.148432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:01:58.148404    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" containerID="cri-o://e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7" gracePeriod=30
Nov 01 01:02:00.579536 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.579500    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:00.579936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.579577    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:00.659383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.659345    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:00.659539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.659410    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:00.812528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.812489    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:00.812713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:00.812570    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:01.795595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.795524    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 01:02:01.896290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896252    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896316    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896427    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896460    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896594 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896496    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896662 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896627    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896698    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.896770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896622    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:01.896770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896701    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:01.896992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896970    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:01.897069 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.896995    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.897069 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897055    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\") pod \"7693e327-337d-4a3f-ad05-1cd9a02796cb\" (UID: \"7693e327-337d-4a3f-ad05-1cd9a02796cb\") "
Nov 01 01:02:01.897306 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897286    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:01.897515 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897498    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:01.897746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897730    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.897785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897751    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.897785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897761    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.897785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897772    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7693e327-337d-4a3f-ad05-1cd9a02796cb-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.897785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.897780    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.899220 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.899194    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:01.899311 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.899237    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:01.899378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.899355    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:01.899418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.899373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47" pod "7693e327-337d-4a3f-ad05-1cd9a02796cb" (UID: "7693e327-337d-4a3f-ad05-1cd9a02796cb"). InnerVolumeSpecName "kube-api-access-m4c47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:01.997770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.997728    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:01.997965 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.997797    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:01.998857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.998829    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4c47\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-kube-api-access-m4c47\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.998857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.998857    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.999036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.998867    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:01.999036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:01.998875    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7693e327-337d-4a3f-ad05-1cd9a02796cb-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:02.536451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.536419    4493 generic.go:356] "Generic (PLEG): container finished" podID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerID="d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2" exitCode=0
Nov 01 01:02:02.536627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.536463    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"
Nov 01 01:02:02.536627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.536507    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" event={"ID":"7693e327-337d-4a3f-ad05-1cd9a02796cb","Type":"ContainerDied","Data":"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"}
Nov 01 01:02:02.536627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.536565    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m" event={"ID":"7693e327-337d-4a3f-ad05-1cd9a02796cb","Type":"ContainerDied","Data":"d215878ee1109102ae3e36685ca0b3be9130ab1b79d3fdf508b29e8a7afe9513"}
Nov 01 01:02:02.536627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.536591    4493 scope.go:122] "RemoveContainer" containerID="d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"
Nov 01 01:02:02.547014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.546995    4493 scope.go:122] "RemoveContainer" containerID="d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"
Nov 01 01:02:02.547424 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:02.547403    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2\": container with ID starting with d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2 not found: ID does not exist" containerID="d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"
Nov 01 01:02:02.547486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.547432    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2"} err="failed to get container status \"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2\": rpc error: code = NotFound desc = could not find container \"d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2\": container with ID starting with d65a75d6a7bc7c16987e14fd053addff3a6fb2e689c3d670c5e58ee4d65213d2 not found: ID does not exist"
Nov 01 01:02:02.713169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.708125    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 01:02:02.755967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:02.755932    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-mwx2m"]
Nov 01 01:02:03.106181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.105792    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 01:02:03.118779 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.118753    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 01:02:03.130436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.130410    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 01:02:03.242622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.242594    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.242798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.242643    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.242798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.242680    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.242970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.242927    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243043 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243021    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:03.243371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243346    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.243477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243433    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243466    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.243610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243595    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243622    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243634    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.243748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243652    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243690    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.243748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243724    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.243894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243762    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.243894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243805    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.243894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243856    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.243894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243887    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.244091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243914    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.244091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.243971    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.244091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244019    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.244296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244099    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.244296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244158    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.244296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244224    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\") pod \"a3b84f89-813b-451d-9255-e0415d0dc81e\" (UID: \"a3b84f89-813b-451d-9255-e0415d0dc81e\") "
Nov 01 01:02:03.244296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244265    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.244481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244303    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.244481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244360    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.244481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244416    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.244481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244453    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\") pod \"0fdab793-77bc-4daa-aefd-80b03bd3277f\" (UID: \"0fdab793-77bc-4daa-aefd-80b03bd3277f\") "
Nov 01 01:02:03.244684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244480    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\") pod \"96c801fe-588b-4c9d-8f42-7c32f2556346\" (UID: \"96c801fe-588b-4c9d-8f42-7c32f2556346\") "
Nov 01 01:02:03.244684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.244539    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.245189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245161    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:03.245405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245380    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245408    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245431    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245449    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c801fe-588b-4c9d-8f42-7c32f2556346-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245461    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245475    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b84f89-813b-451d-9255-e0415d0dc81e-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.245866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.245741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.246091    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.246289    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.246565    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.246754    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.246943    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.247116    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.247194    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.247647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.247251    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.248674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.248014    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:03.251746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.251583    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.251746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.251622    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "kube-api-access-zth5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.251746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.251704    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.252814 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.252781    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy" pod "96c801fe-588b-4c9d-8f42-7c32f2556346" (UID: "96c801fe-588b-4c9d-8f42-7c32f2556346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.253081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.253060    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.253258 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.253218    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.253368 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.253349    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.253464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.253441    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:03.253686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.253633    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:03.254252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.254220    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2" pod "0fdab793-77bc-4daa-aefd-80b03bd3277f" (UID: "0fdab793-77bc-4daa-aefd-80b03bd3277f"). InnerVolumeSpecName "kube-api-access-tgpp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.255824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.255804    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt" pod "a3b84f89-813b-451d-9255-e0415d0dc81e" (UID: "a3b84f89-813b-451d-9255-e0415d0dc81e"). InnerVolumeSpecName "kube-api-access-8vbkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.363787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363752    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.363787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363786    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363801    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbkt\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-kube-api-access-8vbkt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363817    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363846    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363859    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363875    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fdab793-77bc-4daa-aefd-80b03bd3277f-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363889    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363901    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363915    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363932    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363947    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b84f89-813b-451d-9255-e0415d0dc81e-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363962    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zth5s\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-kube-api-access-zth5s\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363975    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c801fe-588b-4c9d-8f42-7c32f2556346-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.363988    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364001    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fdab793-77bc-4daa-aefd-80b03bd3277f-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364015    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364030    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c801fe-588b-4c9d-8f42-7c32f2556346-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364044    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgpp2\" (UniqueName: \"kubernetes.io/projected/0fdab793-77bc-4daa-aefd-80b03bd3277f-kube-api-access-tgpp2\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364059    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b84f89-813b-451d-9255-e0415d0dc81e-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.364127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.364073    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fdab793-77bc-4daa-aefd-80b03bd3277f-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.426830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.426800    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 01:02:03.509411 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509373    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.509816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509452    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.509816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509486    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.509816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509762    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.509816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509805    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.510149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.509956    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.510149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.510033    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.510149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.510098    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.510297 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.510173    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\") pod \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\" (UID: \"cfb4879f-f720-4a9d-b9a1-415fdff20edf\") "
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.523447    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.526483    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.532731    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.532758    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.544644    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.570823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.545860    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.573306 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.573264    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.581519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.581483    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:03.583073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.583047    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.585183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.585157    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "kube-api-access-kfgcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.585672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.585635    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy" pod "cfb4879f-f720-4a9d-b9a1-415fdff20edf" (UID: "cfb4879f-f720-4a9d-b9a1-415fdff20edf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.586343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.586250    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 01:02:03.596181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.587026    4493 generic.go:356] "Generic (PLEG): container finished" podID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerID="72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066" exitCode=0
Nov 01 01:02:03.596181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.590818    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" event={"ID":"96c801fe-588b-4c9d-8f42-7c32f2556346","Type":"ContainerDied","Data":"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"}
Nov 01 01:02:03.596181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.590902    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn" event={"ID":"96c801fe-588b-4c9d-8f42-7c32f2556346","Type":"ContainerDied","Data":"7648a524b04b6591b28e79a4cc8ac7880d390b939766b0fcc012f89d753654d7"}
Nov 01 01:02:03.596181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.590949    4493 scope.go:122] "RemoveContainer" containerID="72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"
Nov 01 01:02:03.596181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.591102    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"
Nov 01 01:02:03.599534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.599506    4493 generic.go:356] "Generic (PLEG): container finished" podID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerID="68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91" exitCode=0
Nov 01 01:02:03.599641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.599610    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"
Nov 01 01:02:03.599641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.599625    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" event={"ID":"cfb4879f-f720-4a9d-b9a1-415fdff20edf","Type":"ContainerDied","Data":"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"}
Nov 01 01:02:03.599751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.599676    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp" event={"ID":"cfb4879f-f720-4a9d-b9a1-415fdff20edf","Type":"ContainerDied","Data":"a59f208fee85389532f5b28df6e82d869a6dee47133a78ed3a315be575703e37"}
Nov 01 01:02:03.602582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.601796    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerID="9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c" exitCode=0
Nov 01 01:02:03.602582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.601822    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerDied","Data":"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"}
Nov 01 01:02:03.602582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.601869    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w" event={"ID":"a3b84f89-813b-451d-9255-e0415d0dc81e","Type":"ContainerDied","Data":"2f7c762a798a2a5509269ac3edf1008b9eefcac7981ca39502d358c714496181"}
Nov 01 01:02:03.602582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.601964    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"
Nov 01 01:02:03.605105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.605082    4493 generic.go:356] "Generic (PLEG): container finished" podID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerID="b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01" exitCode=0
Nov 01 01:02:03.605244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.605157    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" event={"ID":"0fdab793-77bc-4daa-aefd-80b03bd3277f","Type":"ContainerDied","Data":"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"}
Nov 01 01:02:03.605244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.605173    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"
Nov 01 01:02:03.605244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.605201    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6" event={"ID":"0fdab793-77bc-4daa-aefd-80b03bd3277f","Type":"ContainerDied","Data":"6aea9a911412c7f49a6562dcd1974527e2a5510691028e2dbdf1b6af816bd239"}
Nov 01 01:02:03.612450 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.612420    4493 generic.go:356] "Generic (PLEG): container finished" podID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerID="e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7" exitCode=0
Nov 01 01:02:03.612546 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.612514    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" event={"ID":"cfddc601-9e1f-452e-bb4f-9e3441f35902","Type":"ContainerDied","Data":"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"}
Nov 01 01:02:03.612624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.612556    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64" event={"ID":"cfddc601-9e1f-452e-bb4f-9e3441f35902","Type":"ContainerDied","Data":"3210b58381bb30df46e88ddcccb2110ad6f7c16bb4ec4172f8b41a84b7f804f7"}
Nov 01 01:02:03.612696 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.612630    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"
Nov 01 01:02:03.620390 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.620371    4493 scope.go:122] "RemoveContainer" containerID="72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"
Nov 01 01:02:03.621057 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.621038    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066\": container with ID starting with 72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066 not found: ID does not exist" containerID="72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"
Nov 01 01:02:03.621193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.621063    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066"} err="failed to get container status \"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066\": rpc error: code = NotFound desc = could not find container \"72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066\": container with ID starting with 72866a62fa84345a2c35a525cdca05725f4d72927e1eda113d1dae3104c9f066 not found: ID does not exist"
Nov 01 01:02:03.621193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.621081    4493 scope.go:122] "RemoveContainer" containerID="68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"
Nov 01 01:02:03.634965 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.634934    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgcq\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-kube-api-access-kfgcq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.634967    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.634984    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.634997    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.635010    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb4879f-f720-4a9d-b9a1-415fdff20edf-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.635022    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.635085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.635035    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb4879f-f720-4a9d-b9a1-415fdff20edf-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.636248 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.636106    4493 scope.go:122] "RemoveContainer" containerID="68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"
Nov 01 01:02:03.639330 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.639288    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91\": container with ID starting with 68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91 not found: ID does not exist" containerID="68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"
Nov 01 01:02:03.639425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.639343    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91"} err="failed to get container status \"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91\": rpc error: code = NotFound desc = could not find container \"68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91\": container with ID starting with 68f815f140887cc00226bcaecca6a4cb1fb7eda0f8f36eea9a5114e7ed349a91 not found: ID does not exist"
Nov 01 01:02:03.639425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.639368    4493 scope.go:122] "RemoveContainer" containerID="9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"
Nov 01 01:02:03.656670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.656647    4493 scope.go:122] "RemoveContainer" containerID="dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"
Nov 01 01:02:03.666691 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.666670    4493 scope.go:122] "RemoveContainer" containerID="9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"
Nov 01 01:02:03.666978 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.666955    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c\": container with ID starting with 9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c not found: ID does not exist" containerID="9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"
Nov 01 01:02:03.667085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.666988    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c"} err="failed to get container status \"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c\": rpc error: code = NotFound desc = could not find container \"9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c\": container with ID starting with 9c14c78a4b2d41058520341953e51183a88b1ccc9cf7cdaa869b7348ef85045c not found: ID does not exist"
Nov 01 01:02:03.667085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.667014    4493 scope.go:122] "RemoveContainer" containerID="dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"
Nov 01 01:02:03.667371 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.667348    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887\": container with ID starting with dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887 not found: ID does not exist" containerID="dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"
Nov 01 01:02:03.667453 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.667380    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887"} err="failed to get container status \"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887\": rpc error: code = NotFound desc = could not find container \"dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887\": container with ID starting with dc4b6aa3408eef43e0ef42880d3262421d3873e507fdbb93f2104c53657fb887 not found: ID does not exist"
Nov 01 01:02:03.667453 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.667401    4493 scope.go:122] "RemoveContainer" containerID="b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"
Nov 01 01:02:03.679678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.679653    4493 scope.go:122] "RemoveContainer" containerID="b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"
Nov 01 01:02:03.679992 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.679971    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01\": container with ID starting with b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01 not found: ID does not exist" containerID="b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"
Nov 01 01:02:03.680057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.680001    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01"} err="failed to get container status \"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01\": rpc error: code = NotFound desc = could not find container \"b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01\": container with ID starting with b699f05c8033729ad1a695d0f9022f4fe9f278206c17107e9f5c0305d695aa01 not found: ID does not exist"
Nov 01 01:02:03.680057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.680019    4493 scope.go:122] "RemoveContainer" containerID="e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"
Nov 01 01:02:03.692179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.692162    4493 scope.go:122] "RemoveContainer" containerID="e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"
Nov 01 01:02:03.692491 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:03.692468    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7\": container with ID starting with e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7 not found: ID does not exist" containerID="e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"
Nov 01 01:02:03.692565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.692500    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7"} err="failed to get container status \"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7\": rpc error: code = NotFound desc = could not find container \"e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7\": container with ID starting with e28838d4c8fa10000d747095c59afc18133268abe5ad905e17ffc0cd4fdcbbc7 not found: ID does not exist"
Nov 01 01:02:03.736246 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736222    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736259    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736291    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736450    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736482    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736505    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.736572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736526    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736531    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.736572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736555    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736598    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736651    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\") pod \"cfddc601-9e1f-452e-bb4f-9e3441f35902\" (UID: \"cfddc601-9e1f-452e-bb4f-9e3441f35902\") "
Nov 01 01:02:03.736776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736709    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.736933 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.736814    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:03.737039 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737015    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.737503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737480    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.737625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737508    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.737625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737524    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.737625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737538    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfddc601-9e1f-452e-bb4f-9e3441f35902-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.737625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.737552    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.738636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.738614    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.738823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.738796    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:03.738823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.738806    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:03.738983 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.738964    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz" pod "cfddc601-9e1f-452e-bb4f-9e3441f35902" (UID: "cfddc601-9e1f-452e-bb4f-9e3441f35902"). InnerVolumeSpecName "kube-api-access-dq6jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:03.754302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.754271    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 01:02:03.757722 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.757691    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-8gxvp"]
Nov 01 01:02:03.839046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.839008    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.839046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.839041    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6jz\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-kube-api-access-dq6jz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.839279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.839059    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.839279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.839069    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfddc601-9e1f-452e-bb4f-9e3441f35902-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:03.864576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.864479    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 01:02:03.867983 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.867964    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-hr6j6"]
Nov 01 01:02:03.978263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.978225    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 01:02:03.982100 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:03.982072    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-r45pn"]
Nov 01 01:02:04.091293 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.091215    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 01:02:04.095121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.095096    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-lpm9w"]
Nov 01 01:02:04.333865 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.333834    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 01:02:04.338748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.338725    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-7bx64"]
Nov 01 01:02:04.505002 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.504962    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" path="/var/lib/kubelet/pods/0fdab793-77bc-4daa-aefd-80b03bd3277f/volumes"
Nov 01 01:02:04.505624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.505601    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" path="/var/lib/kubelet/pods/7693e327-337d-4a3f-ad05-1cd9a02796cb/volumes"
Nov 01 01:02:04.506122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.506102    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" path="/var/lib/kubelet/pods/96c801fe-588b-4c9d-8f42-7c32f2556346/volumes"
Nov 01 01:02:04.506919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.506898    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" path="/var/lib/kubelet/pods/a3b84f89-813b-451d-9255-e0415d0dc81e/volumes"
Nov 01 01:02:04.507470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.507408    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" path="/var/lib/kubelet/pods/cfb4879f-f720-4a9d-b9a1-415fdff20edf/volumes"
Nov 01 01:02:04.508465 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:04.508447    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" path="/var/lib/kubelet/pods/cfddc601-9e1f-452e-bb4f-9e3441f35902/volumes"
Nov 01 01:02:08.644467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:08.644434    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 01:02:08.644942 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:08.644689    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" containerID="cri-o://5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948" gracePeriod=30
Nov 01 01:02:12.047056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.047009    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:12.047449 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.047071    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:12.504490 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.504461    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 01:02:12.504677 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.504654    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" containerID="cri-o://4c8f7dbf0c86b01052f3958ec6b54502fe7036f8e28bb872c3d46477e4efe261" gracePeriod=30
Nov 01 01:02:12.552071 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.552034    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 01:02:12.552270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:12.552249    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" containerID="cri-o://bedb056c3dbceaacfb724d767f1391d9fade4bdeddd3c53ae2af19ed4ccd7dfc" gracePeriod=30
Nov 01 01:02:13.913840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.913772    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 01:02:13.991769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.991734    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.991950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.991836    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.991950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.991880    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.991950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.991911    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992110    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992224 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992082    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:13.992224 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992191    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:13.992336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992227    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992284    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992317    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992398    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\") pod \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\" (UID: \"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3\") "
Nov 01 01:02:13.992532 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992497    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:13.992703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992667    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:13.992785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992724    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:13.992990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992972    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:13.993049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.992993    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:13.993049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.993003    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:13.993049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.993011    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:13.993049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.993019    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:13.994931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.994905    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:13.994931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.994916    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:13.995083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.994939    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:13.995083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:13.994988    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt" pod "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" (UID: "1f1e12db-6952-45d4-9cfc-609e9e8dd4f3"). InnerVolumeSpecName "kube-api-access-zzsqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:14.094810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.094777    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzsqt\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-kube-api-access-zzsqt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:14.094810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.094806    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:14.095007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.094828    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:14.095007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.094837    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:14.363983 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.363947    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 01:02:14.477972 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.476973    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 01:02:14.763348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.763306    4493 generic.go:356] "Generic (PLEG): container finished" podID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerID="5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948" exitCode=0
Nov 01 01:02:14.763527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.763382    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"
Nov 01 01:02:14.763527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.763389    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" event={"ID":"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3","Type":"ContainerDied","Data":"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"}
Nov 01 01:02:14.763527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.763448    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r" event={"ID":"1f1e12db-6952-45d4-9cfc-609e9e8dd4f3","Type":"ContainerDied","Data":"3ba93dc0ef864faf2ac069bb9fe61b80909f46a37b9c829e974cc1c5bea9ecee"}
Nov 01 01:02:14.763527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.763470    4493 scope.go:122] "RemoveContainer" containerID="5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"
Nov 01 01:02:14.814169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.803340    4493 scope.go:122] "RemoveContainer" containerID="5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"
Nov 01 01:02:14.814169 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:14.805567    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948\": container with ID starting with 5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948 not found: ID does not exist" containerID="5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"
Nov 01 01:02:14.814169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.805609    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948"} err="failed to get container status \"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948\": rpc error: code = NotFound desc = could not find container \"5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948\": container with ID starting with 5ae3ebfe4b0a4eeb5ce8d09cc4dc585f107fbddaa170995ae3529369650c0948 not found: ID does not exist"
Nov 01 01:02:14.963394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.963359    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 01:02:14.966977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:14.966947    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-gz74r"]
Nov 01 01:02:14.995791 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:14.995761    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:02:15.954936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:15.954897    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:15.955185 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:15.954961    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:16.504362 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:16.504333    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" path="/var/lib/kubelet/pods/1f1e12db-6952-45d4-9cfc-609e9e8dd4f3/volumes"
Nov 01 01:02:17.412208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.412175    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:02:17.412457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.412231    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:02:17.786586 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.786557    4493 generic.go:356] "Generic (PLEG): container finished" podID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerID="bedb056c3dbceaacfb724d767f1391d9fade4bdeddd3c53ae2af19ed4ccd7dfc" exitCode=0
Nov 01 01:02:17.787023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.786634    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" event={"ID":"bc012d25-c8fa-42ee-8e55-a05e2606cd87","Type":"ContainerDied","Data":"bedb056c3dbceaacfb724d767f1391d9fade4bdeddd3c53ae2af19ed4ccd7dfc"}
Nov 01 01:02:17.788475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.788444    4493 generic.go:356] "Generic (PLEG): container finished" podID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerID="4c8f7dbf0c86b01052f3958ec6b54502fe7036f8e28bb872c3d46477e4efe261" exitCode=0
Nov 01 01:02:17.788602 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.788498    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" event={"ID":"ce48cca2-61fa-4da0-b543-3eaf0d86af79","Type":"ContainerDied","Data":"4c8f7dbf0c86b01052f3958ec6b54502fe7036f8e28bb872c3d46477e4efe261"}
Nov 01 01:02:17.844114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.844091    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 01:02:17.861967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.861934    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862072    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862168    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862209    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862239    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862304    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862383    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862486    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862526    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\") pod \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\" (UID: \"ce48cca2-61fa-4da0-b543-3eaf0d86af79\") "
Nov 01 01:02:17.862634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862529    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.863159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.862781    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.863159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.863159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863083    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.863159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863118    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:17.863440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863273    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.863440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863293    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.863440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863310    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.863440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863332    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.863440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.863345    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.865768 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.865741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:17.865940 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.865853    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.866024 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.865945    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "kube-api-access-rddz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:17.866719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.866696    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token" pod "ce48cca2-61fa-4da0-b543-3eaf0d86af79" (UID: "ce48cca2-61fa-4da0-b543-3eaf0d86af79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:17.884939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.884903    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 01:02:17.964292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964218    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964307    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964336    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964436    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964468    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964509    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964542    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964576    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964619    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\") pod \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\" (UID: \"bc012d25-c8fa-42ee-8e55-a05e2606cd87\") "
Nov 01 01:02:17.964925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964751    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.964925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964766    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.964925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964881    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:17.964925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.964890    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.965166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965149    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.965547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965522    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965552    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965569    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965583    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965598    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965613    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965629    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce48cca2-61fa-4da0-b543-3eaf0d86af79-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965643    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.965920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.965659    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddz2\" (UniqueName: \"kubernetes.io/projected/ce48cca2-61fa-4da0-b543-3eaf0d86af79-kube-api-access-rddz2\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:17.966659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.966632    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:02:17.966804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.966784    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:02:17.966890 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.966863    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "kube-api-access-zvbt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:17.966987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:17.966936    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token" pod "bc012d25-c8fa-42ee-8e55-a05e2606cd87" (UID: "bc012d25-c8fa-42ee-8e55-a05e2606cd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:18.067302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.067263    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:18.067302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.067294    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:18.067302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.067308    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc012d25-c8fa-42ee-8e55-a05e2606cd87-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:18.067550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.067328    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbt4\" (UniqueName: \"kubernetes.io/projected/bc012d25-c8fa-42ee-8e55-a05e2606cd87-kube-api-access-zvbt4\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:18.796340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.796253    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"
Nov 01 01:02:18.796340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.796273    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8" event={"ID":"ce48cca2-61fa-4da0-b543-3eaf0d86af79","Type":"ContainerDied","Data":"9a72634bf96e8daf2430fe7d3b04e54e169e50eef68477f49e08c0a4398cc735"}
Nov 01 01:02:18.796340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.796334    4493 scope.go:122] "RemoveContainer" containerID="4c8f7dbf0c86b01052f3958ec6b54502fe7036f8e28bb872c3d46477e4efe261"
Nov 01 01:02:18.798058 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.798032    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58" event={"ID":"bc012d25-c8fa-42ee-8e55-a05e2606cd87","Type":"ContainerDied","Data":"019d2d7c3ba2ef066079116e96127b074404c2bbb3b6c3e8077d482ad5fff01b"}
Nov 01 01:02:18.798214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.798160    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"
Nov 01 01:02:18.806849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:18.806834    4493 scope.go:122] "RemoveContainer" containerID="bedb056c3dbceaacfb724d767f1391d9fade4bdeddd3c53ae2af19ed4ccd7dfc"
Nov 01 01:02:19.013817 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.013782    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 01:02:19.063505 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.063428    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xc9c8"]
Nov 01 01:02:19.127010 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.126979    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 01:02:19.131308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.131280    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-xwr58"]
Nov 01 01:02:19.364899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.364816    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" podUID="b088c408-387d-4f89-b320-ec70e5754808" containerName="payload-processing" containerID="cri-o://0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048" gracePeriod=25
Nov 01 01:02:19.477887 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.477850    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" podUID="d70360ad-5b31-4781-8603-2332f23c9d6c" containerName="payload-pre-processing" containerID="cri-o://57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c" gracePeriod=25
Nov 01 01:02:19.650020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.649994    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 01:02:19.688289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.688251    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\" (UniqueName: \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\") pod \"b088c408-387d-4f89-b320-ec70e5754808\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") "
Nov 01 01:02:19.688467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.688345    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\") pod \"b088c408-387d-4f89-b320-ec70e5754808\" (UID: \"b088c408-387d-4f89-b320-ec70e5754808\") "
Nov 01 01:02:19.688794 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.688762    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume" pod "b088c408-387d-4f89-b320-ec70e5754808" (UID: "b088c408-387d-4f89-b320-ec70e5754808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:19.689187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.689165    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b088c408-387d-4f89-b320-ec70e5754808-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:19.690940 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.690915    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x" pod "b088c408-387d-4f89-b320-ec70e5754808" (UID: "b088c408-387d-4f89-b320-ec70e5754808"). InnerVolumeSpecName "kube-api-access-q989x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:19.798164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.797768    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 01:02:19.843154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.822641    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q989x\" (UniqueName: \"kubernetes.io/projected/b088c408-387d-4f89-b320-ec70e5754808-kube-api-access-q989x\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:19.886329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.885989    4493 generic.go:356] "Generic (PLEG): container finished" podID="d70360ad-5b31-4781-8603-2332f23c9d6c" containerID="57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c" exitCode=0
Nov 01 01:02:19.886329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.886122    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" event={"ID":"d70360ad-5b31-4781-8603-2332f23c9d6c","Type":"ContainerDied","Data":"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"}
Nov 01 01:02:19.886329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.886181    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck" event={"ID":"d70360ad-5b31-4781-8603-2332f23c9d6c","Type":"ContainerDied","Data":"253f61c7873b11371d464ff136dedae89bc582abed4f01a194be23de016ca5a7"}
Nov 01 01:02:19.886329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.886210    4493 scope.go:122] "RemoveContainer" containerID="57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"
Nov 01 01:02:19.886329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.886288    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"
Nov 01 01:02:19.891011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.889871    4493 generic.go:356] "Generic (PLEG): container finished" podID="b088c408-387d-4f89-b320-ec70e5754808" containerID="0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048" exitCode=0
Nov 01 01:02:19.891011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.890049    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" event={"ID":"b088c408-387d-4f89-b320-ec70e5754808","Type":"ContainerDied","Data":"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"}
Nov 01 01:02:19.891011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.890097    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4" event={"ID":"b088c408-387d-4f89-b320-ec70e5754808","Type":"ContainerDied","Data":"6f653cb9271a41cf612526d9b36e6e510e17719f585f10eeb40727ca81e0cd2b"}
Nov 01 01:02:19.891011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.890186    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"
Nov 01 01:02:19.902755 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.902729    4493 scope.go:122] "RemoveContainer" containerID="57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"
Nov 01 01:02:19.903075 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:19.903043    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c\": container with ID starting with 57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c not found: ID does not exist" containerID="57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"
Nov 01 01:02:19.903172 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.903087    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c"} err="failed to get container status \"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c\": rpc error: code = NotFound desc = could not find container \"57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c\": container with ID starting with 57c75ce697d94f14e51ef6dea0992c0249de120308ccb32c751f3ea31e8b137c not found: ID does not exist"
Nov 01 01:02:19.903172 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.903111    4493 scope.go:122] "RemoveContainer" containerID="0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.924353    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\" (UniqueName: \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\") pod \"d70360ad-5b31-4781-8603-2332f23c9d6c\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") "
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.924406    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\") pod \"d70360ad-5b31-4781-8603-2332f23c9d6c\" (UID: \"d70360ad-5b31-4781-8603-2332f23c9d6c\") "
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.925665    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume" pod "d70360ad-5b31-4781-8603-2332f23c9d6c" (UID: "d70360ad-5b31-4781-8603-2332f23c9d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.925915    4493 scope.go:122] "RemoveContainer" containerID="0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:19.926423    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048\": container with ID starting with 0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048 not found: ID does not exist" containerID="0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.926460    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048"} err="failed to get container status \"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048\": rpc error: code = NotFound desc = could not find container \"0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048\": container with ID starting with 0d1decd4f101b28398f7bdd5fd371de9ea11efbfe92433e9b4f9c6401f362048 not found: ID does not exist"
Nov 01 01:02:19.929091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:19.928121    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc" pod "d70360ad-5b31-4781-8603-2332f23c9d6c" (UID: "d70360ad-5b31-4781-8603-2332f23c9d6c"). InnerVolumeSpecName "kube-api-access-c7mmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:20.026550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.026503    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mmc\" (UniqueName: \"kubernetes.io/projected/d70360ad-5b31-4781-8603-2332f23c9d6c-kube-api-access-c7mmc\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:20.026550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.026543    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70360ad-5b31-4781-8603-2332f23c9d6c-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:20.029549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.029514    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 01:02:20.033237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.033216    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-27dee6-bb87468d4-7p8s4"]
Nov 01 01:02:20.307848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.307817    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 01:02:20.312556 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.312527    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-27dee6-6c4dcb6c9f-pjvck"]
Nov 01 01:02:20.503759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.503726    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b088c408-387d-4f89-b320-ec70e5754808" path="/var/lib/kubelet/pods/b088c408-387d-4f89-b320-ec70e5754808/volumes"
Nov 01 01:02:20.504265 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.504250    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" path="/var/lib/kubelet/pods/bc012d25-c8fa-42ee-8e55-a05e2606cd87/volumes"
Nov 01 01:02:20.504670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.504658    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" path="/var/lib/kubelet/pods/ce48cca2-61fa-4da0-b543-3eaf0d86af79/volumes"
Nov 01 01:02:20.505355 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:20.505341    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70360ad-5b31-4781-8603-2332f23c9d6c" path="/var/lib/kubelet/pods/d70360ad-5b31-4781-8603-2332f23c9d6c/volumes"
Nov 01 01:02:24.553970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:24.553935    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 01:02:25.353717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:25.353683    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:02:29.554248 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.554200    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" podUID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" containerName="payload-processing" containerID="cri-o://f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf" gracePeriod=25
Nov 01 01:02:29.825875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.825812    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 01:02:29.898700 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.898671    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\" (UniqueName: \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\") pod \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") "
Nov 01 01:02:29.898880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.898828    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\") pod \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\" (UID: \"3fab8f56-4cb0-47ab-bcf6-48e501d42f93\") "
Nov 01 01:02:29.899394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.899360    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume" pod "3fab8f56-4cb0-47ab-bcf6-48e501d42f93" (UID: "3fab8f56-4cb0-47ab-bcf6-48e501d42f93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:02:29.899700 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.899680    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:29.901435 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:29.901412    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q" pod "3fab8f56-4cb0-47ab-bcf6-48e501d42f93" (UID: "3fab8f56-4cb0-47ab-bcf6-48e501d42f93"). InnerVolumeSpecName "kube-api-access-pvw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:02:30.000051 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.000002    4493 generic.go:356] "Generic (PLEG): container finished" podID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" containerID="f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf" exitCode=0
Nov 01 01:02:30.000282 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.000081    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"
Nov 01 01:02:30.000282 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.000272    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" event={"ID":"3fab8f56-4cb0-47ab-bcf6-48e501d42f93","Type":"ContainerDied","Data":"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"}
Nov 01 01:02:30.000394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.000362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8" event={"ID":"3fab8f56-4cb0-47ab-bcf6-48e501d42f93","Type":"ContainerDied","Data":"72a8157b01c95d84b3cd6e8c46d497c8bc0662faa185cb99e45c4541788cff71"}
Nov 01 01:02:30.000445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.000392    4493 scope.go:122] "RemoveContainer" containerID="f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"
Nov 01 01:02:30.001085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.001055    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvw7q\" (UniqueName: \"kubernetes.io/projected/3fab8f56-4cb0-47ab-bcf6-48e501d42f93-kube-api-access-pvw7q\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:02:30.011383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.011364    4493 scope.go:122] "RemoveContainer" containerID="f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"
Nov 01 01:02:30.011684 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:30.011664    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf\": container with ID starting with f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf not found: ID does not exist" containerID="f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"
Nov 01 01:02:30.011754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.011697    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf"} err="failed to get container status \"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf\": rpc error: code = NotFound desc = could not find container \"f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf\": container with ID starting with f626dab15e0367af48029ce0bb650e1c9524b68327a8ba3cbb19e7c13e6db3cf not found: ID does not exist"
Nov 01 01:02:30.166404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.166322    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 01:02:30.189940 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.171117    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af82f1-94688f944-q75g8"]
Nov 01 01:02:30.504392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:30.504362    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" path="/var/lib/kubelet/pods/3fab8f56-4cb0-47ab-bcf6-48e501d42f93/volumes"
Nov 01 01:02:41.882979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.882947    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:02:41.883640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883623    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883643    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883653    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b088c408-387d-4f89-b320-ec70e5754808" containerName="payload-processing"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883658    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b088c408-387d-4f89-b320-ec70e5754808" containerName="payload-processing"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883671    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" containerName="payload-processing"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883676    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" containerName="payload-processing"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883684    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy"
Nov 01 01:02:41.883701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883689    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883708    4493 cpu_manager.go:414] "containerMap: removing container" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883716    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883726    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883731    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883739    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d70360ad-5b31-4781-8603-2332f23c9d6c" containerName="payload-pre-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883744    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70360ad-5b31-4781-8603-2332f23c9d6c" containerName="payload-pre-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883751    4493 cpu_manager.go:414] "containerMap: removing container" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883756    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883762    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883766    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883779    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883784    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883795    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883802    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883815    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883819    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883827    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883831    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883840    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883845    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883854    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy"
Nov 01 01:02:41.883920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883858    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883956    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7693e327-337d-4a3f-ad05-1cd9a02796cb" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883967    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b088c408-387d-4f89-b320-ec70e5754808" containerName="payload-processing"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883972    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fab8f56-4cb0-47ab-bcf6-48e501d42f93" containerName="payload-processing"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883979    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdab793-77bc-4daa-aefd-80b03bd3277f" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883987    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c801fe-588b-4c9d-8f42-7c32f2556346" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883992    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec90e9f-68ab-49cd-85de-4521d65aff26" containerName="payload-processing"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.883997    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70360ad-5b31-4781-8603-2332f23c9d6c" containerName="payload-pre-processing"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884003    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="722513e5-97ca-41c3-95df-6cddd56cb1d7" containerName="payload-pre-processing"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884009    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfddc601-9e1f-452e-bb4f-9e3441f35902" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884015    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc012d25-c8fa-42ee-8e55-a05e2606cd87" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884021    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884030    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb4879f-f720-4a9d-b9a1-415fdff20edf" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884037    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b84f89-813b-451d-9255-e0415d0dc81e" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884043    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce48cca2-61fa-4da0-b543-3eaf0d86af79" containerName="istio-proxy"
Nov 01 01:02:41.884592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.884048    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1e12db-6952-45d4-9cfc-609e9e8dd4f3" containerName="istio-proxy"
Nov 01 01:02:41.889548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.889531    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:41.942162 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.942102    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f73fb0-serving-cert\""
Nov 01 01:02:41.947293 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.947261    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:02:41.967963 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.967927    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:41.968121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.967972    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:41.968194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.968122    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmsfp\" (UniqueName: \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:41.968291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:41.968272    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.069631 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.069597    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.069817 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.069720    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.069817 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:42.069752    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-serving-cert: secret "maas-api-e2e-api-b-f73fb0-serving-cert" not found
Nov 01 01:02:42.069817 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.069759    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.069817 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:42.069816    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls podName:edf52fd3-5f28-495b-ac4e-e100fc3bf915 nodeName:}" failed. No retries permitted until 2026-11-01 01:02:42.569795357 +0000 UTC m=+2352.247382303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls") pod "maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" (UID: "edf52fd3-5f28-495b-ac4e-e100fc3bf915") : secret "maas-api-e2e-api-b-f73fb0-serving-cert" not found
Nov 01 01:02:42.070037 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.069896    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmsfp\" (UniqueName: \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.070772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.070740    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.075554 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.075520    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.133716 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.133642    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmsfp\" (UniqueName: \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.581632 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.581591    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.584416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.584390    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") pod \"maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:42.803184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:42.803119    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:43.182802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:43.182772    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:02:43.183963 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:02:43.183934    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf52fd3_5f28_495b_ac4e_e100fc3bf915.slice/crio-81e1fb942a4e28be48f3b2b04e2ddedecc1d847af793df213d65d1b82ec83570 WatchSource:0}: Error finding container 81e1fb942a4e28be48f3b2b04e2ddedecc1d847af793df213d65d1b82ec83570: Status 404 returned error can't find the container with id 81e1fb942a4e28be48f3b2b04e2ddedecc1d847af793df213d65d1b82ec83570
Nov 01 01:02:44.091190 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:44.091165    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" event={"ID":"edf52fd3-5f28-495b-ac4e-e100fc3bf915","Type":"ContainerStarted","Data":"81e1fb942a4e28be48f3b2b04e2ddedecc1d847af793df213d65d1b82ec83570"}
Nov 01 01:02:44.990421 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:02:44.990388    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:02:45.099726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:45.099690    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" event={"ID":"edf52fd3-5f28-495b-ac4e-e100fc3bf915","Type":"ContainerStarted","Data":"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"}
Nov 01 01:02:45.099931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:45.099821    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:02:45.219007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:45.218957    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" podStartSLOduration=3.380791706 podStartE2EDuration="4.218938498s" totalImagesPullingTime="838.146792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:02:45.217572974 +0000 UTC m=+2354.895159920" watchObservedRunningTime="2026-11-01 01:02:45.218938498 +0000 UTC m=+2354.896525439"
Nov 01 01:02:51.113221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:02:51.113192    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:03:13.271174 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:13.271082    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:03:14.991352 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:14.991308    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:03:22.688167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:22.688116    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 01:03:22.709165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:22.689074    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" podUID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" containerName="maas-api" containerID="cri-o://e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc" gracePeriod=30
Nov 01 01:03:23.122725 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.122700    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 01:03:23.298980 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.298878    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\" (UniqueName: \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\") pod \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") "
Nov 01 01:03:23.299225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.299007    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\") pod \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") "
Nov 01 01:03:23.299225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.299108    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") pod \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") "
Nov 01 01:03:23.300116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.299654    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\") pod \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\" (UID: \"f8a4dde4-1f43-46d2-8411-eac45411bbf1\") "
Nov 01 01:03:23.301018 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.300990    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca" pod "f8a4dde4-1f43-46d2-8411-eac45411bbf1" (UID: "f8a4dde4-1f43-46d2-8411-eac45411bbf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:23.303265 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.303242    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb" pod "f8a4dde4-1f43-46d2-8411-eac45411bbf1" (UID: "f8a4dde4-1f43-46d2-8411-eac45411bbf1"). InnerVolumeSpecName "kube-api-access-2wvtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:23.303660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.303639    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs" pod "f8a4dde4-1f43-46d2-8411-eac45411bbf1" (UID: "f8a4dde4-1f43-46d2-8411-eac45411bbf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:03:23.303809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.303792    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls" pod "f8a4dde4-1f43-46d2-8411-eac45411bbf1" (UID: "f8a4dde4-1f43-46d2-8411-eac45411bbf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:03:23.435514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.412261    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:23.435514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.412296    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a4dde4-1f43-46d2-8411-eac45411bbf1-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:23.435514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.412311    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wvtb\" (UniqueName: \"kubernetes.io/projected/f8a4dde4-1f43-46d2-8411-eac45411bbf1-kube-api-access-2wvtb\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:23.435514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.412327    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8a4dde4-1f43-46d2-8411-eac45411bbf1-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:23.435890 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.435833    4493 generic.go:356] "Generic (PLEG): container finished" podID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" containerID="e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc" exitCode=0
Nov 01 01:03:23.435946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.435918    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" event={"ID":"f8a4dde4-1f43-46d2-8411-eac45411bbf1","Type":"ContainerDied","Data":"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"}
Nov 01 01:03:23.436007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.435971    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl" event={"ID":"f8a4dde4-1f43-46d2-8411-eac45411bbf1","Type":"ContainerDied","Data":"2514204776fc3b6830181c48bfa044a7f1c0d08fe2313978fcc2835b000a1b73"}
Nov 01 01:03:23.436063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.436007    4493 scope.go:122] "RemoveContainer" containerID="e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"
Nov 01 01:03:23.436213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.436192    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"
Nov 01 01:03:23.448518 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.448483    4493 scope.go:122] "RemoveContainer" containerID="e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"
Nov 01 01:03:23.448795 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:23.448770    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc\": container with ID starting with e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc not found: ID does not exist" containerID="e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"
Nov 01 01:03:23.448858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.448810    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc"} err="failed to get container status \"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc\": rpc error: code = NotFound desc = could not find container \"e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc\": container with ID starting with e5b623d79df1e2ee5d54743a619b5aa60ac13c29e8e06f939fa73166e153b8dc not found: ID does not exist"
Nov 01 01:03:23.603069 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.602992    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 01:03:23.609484 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:23.609459    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af82f1-67f5cd8f84-cgdfl"]
Nov 01 01:03:24.551837 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.551807    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" path="/var/lib/kubelet/pods/f8a4dde4-1f43-46d2-8411-eac45411bbf1/volumes"
Nov 01 01:03:24.571068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.571034    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:03:24.571727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.571713    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" containerName="maas-api"
Nov 01 01:03:24.571776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.571728    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" containerName="maas-api"
Nov 01 01:03:24.571818 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.571809    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a4dde4-1f43-46d2-8411-eac45411bbf1" containerName="maas-api"
Nov 01 01:03:24.576646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.576627    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.680665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.680633    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-72acf4-serving-cert\""
Nov 01 01:03:24.686193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.686164    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:03:24.752339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.752308    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.752523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.752488    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.752621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.752601    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.752687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.752674    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4rx\" (UniqueName: \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.857219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.856672    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.857219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.856767    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.857219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.856850    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4rx\" (UniqueName: \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.857219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.856977    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.857219 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:24.857208    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-serving-cert: secret "maas-api-e2e-inf-a-72acf4-serving-cert" not found
Nov 01 01:03:24.857769 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:24.857285    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls podName:a4b33793-1b57-441b-acd9-a066d16a9b8f nodeName:}" failed. No retries permitted until 2026-11-01 01:03:25.357265998 +0000 UTC m=+2395.034852936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls") pod "maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" (UID: "a4b33793-1b57-441b-acd9-a066d16a9b8f") : secret "maas-api-e2e-inf-a-72acf4-serving-cert" not found
Nov 01 01:03:24.867167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.862248    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.884167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.882904    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:24.949224 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:24.949196    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4rx\" (UniqueName: \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:25.369167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:25.369097    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:25.371676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:25.371656    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") pod \"maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:25.497979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:25.497713    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:26.009550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:26.009520    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:03:26.011098 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:03:26.011068    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b33793_1b57_441b_acd9_a066d16a9b8f.slice/crio-4daf6a484c71fb16cdc6afaf0bb55758a72b5b4a967aa741a5c55a69fc86f297 WatchSource:0}: Error finding container 4daf6a484c71fb16cdc6afaf0bb55758a72b5b4a967aa741a5c55a69fc86f297: Status 404 returned error can't find the container with id 4daf6a484c71fb16cdc6afaf0bb55758a72b5b4a967aa741a5c55a69fc86f297
Nov 01 01:03:26.012820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:26.012799    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 01:03:26.465072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:26.464983    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" event={"ID":"a4b33793-1b57-441b-acd9-a066d16a9b8f","Type":"ContainerStarted","Data":"4daf6a484c71fb16cdc6afaf0bb55758a72b5b4a967aa741a5c55a69fc86f297"}
Nov 01 01:03:27.474092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:27.474040    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" event={"ID":"a4b33793-1b57-441b-acd9-a066d16a9b8f","Type":"ContainerStarted","Data":"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"}
Nov 01 01:03:27.474452 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:27.474247    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:27.594032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:27.593983    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" podStartSLOduration=2.6258646949999997 podStartE2EDuration="3.593967625s" totalImagesPullingTime="968.10293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:03:27.591263192 +0000 UTC m=+2397.268850138" watchObservedRunningTime="2026-11-01 01:03:27.593967625 +0000 UTC m=+2397.271554569"
Nov 01 01:03:32.472081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:32.472049    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 01:03:32.601078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:32.601043    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 01:03:32.645108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:32.645077    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 01:03:32.645302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:32.645282    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy" containerID="cri-o://35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd" gracePeriod=30
Nov 01 01:03:33.486864 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:33.486838    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:03:37.473109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.473066    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" podUID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" containerName="payload-processing" containerID="cri-o://d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33" gracePeriod=25
Nov 01 01:03:37.601498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.601446    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" podUID="19909aa7-543e-480a-acd2-84655e8abb4d" containerName="payload-pre-processing" containerID="cri-o://cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358" gracePeriod=25
Nov 01 01:03:37.770785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.770754    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 01:03:37.905993 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.905958    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\") pod \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") "
Nov 01 01:03:37.906187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.906047    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\" (UniqueName: \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\") pod \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\" (UID: \"6f3100ef-68f3-4199-9aaf-e4ea15262a55\") "
Nov 01 01:03:37.908276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.908246    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume" pod "6f3100ef-68f3-4199-9aaf-e4ea15262a55" (UID: "6f3100ef-68f3-4199-9aaf-e4ea15262a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:37.914060 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.914030    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l" pod "6f3100ef-68f3-4199-9aaf-e4ea15262a55" (UID: "6f3100ef-68f3-4199-9aaf-e4ea15262a55"). InnerVolumeSpecName "kube-api-access-b5w6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:37.918039 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.918014    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 01:03:37.958959 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:37.958936    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 01:03:38.007780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.007713    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\" (UniqueName: \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\") pod \"19909aa7-543e-480a-acd2-84655e8abb4d\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") "
Nov 01 01:03:38.007910 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.007836    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\") pod \"19909aa7-543e-480a-acd2-84655e8abb4d\" (UID: \"19909aa7-543e-480a-acd2-84655e8abb4d\") "
Nov 01 01:03:38.008268 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.008240    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume" pod "19909aa7-543e-480a-acd2-84655e8abb4d" (UID: "19909aa7-543e-480a-acd2-84655e8abb4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:38.008534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.008499    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3100ef-68f3-4199-9aaf-e4ea15262a55-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.008534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.008517    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19909aa7-543e-480a-acd2-84655e8abb4d-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.008656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.008533    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5w6l\" (UniqueName: \"kubernetes.io/projected/6f3100ef-68f3-4199-9aaf-e4ea15262a55-kube-api-access-b5w6l\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.009923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.009902    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z" pod "19909aa7-543e-480a-acd2-84655e8abb4d" (UID: "19909aa7-543e-480a-acd2-84655e8abb4d"). InnerVolumeSpecName "kube-api-access-t2r8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:38.109678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109649    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.109862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109723    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.109862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109745    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.109862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109765    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.109862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109786    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.110082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109882    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.110082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109936    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.110082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.109936    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:03:38.110082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110016    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.110082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110079    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\") pod \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\" (UID: \"3320394e-5d9c-43a9-a2f2-c847ed77f3c5\") "
Nov 01 01:03:38.110385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110075    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:03:38.110385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110167    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:03:38.110385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110179    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:03:38.110702 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110517    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:38.110925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110903    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.110992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110930    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.110992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110945    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.110992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110959    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.110992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110972    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.110992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.110988    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2r8z\" (UniqueName: \"kubernetes.io/projected/19909aa7-543e-480a-acd2-84655e8abb4d-kube-api-access-t2r8z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.112195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.112176    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:03:38.112346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.112327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:38.112694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.112676    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:03:38.112797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.112706    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq" pod "3320394e-5d9c-43a9-a2f2-c847ed77f3c5" (UID: "3320394e-5d9c-43a9-a2f2-c847ed77f3c5"). InnerVolumeSpecName "kube-api-access-dg5lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:38.212302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.212272    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.212302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.212299    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.212302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.212309    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5lq\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-kube-api-access-dg5lq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.212532 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.212319    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3320394e-5d9c-43a9-a2f2-c847ed77f3c5-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:38.544505 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.544462    4493 generic.go:356] "Generic (PLEG): container finished" podID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerID="35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd" exitCode=0
Nov 01 01:03:38.544913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.544525    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"
Nov 01 01:03:38.544913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.544545    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerDied","Data":"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"}
Nov 01 01:03:38.544913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.544597    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr" event={"ID":"3320394e-5d9c-43a9-a2f2-c847ed77f3c5","Type":"ContainerDied","Data":"59913fd4ddd6ea384d4989ecb73314f2aa259225aff59a7ab09241f4ab05b727"}
Nov 01 01:03:38.544913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.544624    4493 scope.go:122] "RemoveContainer" containerID="35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"
Nov 01 01:03:38.546782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.546190    4493 generic.go:356] "Generic (PLEG): container finished" podID="19909aa7-543e-480a-acd2-84655e8abb4d" containerID="cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358" exitCode=0
Nov 01 01:03:38.546782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.546242    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"
Nov 01 01:03:38.546782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.546305    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" event={"ID":"19909aa7-543e-480a-acd2-84655e8abb4d","Type":"ContainerDied","Data":"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"}
Nov 01 01:03:38.546782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.546346    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9" event={"ID":"19909aa7-543e-480a-acd2-84655e8abb4d","Type":"ContainerDied","Data":"9ee7637a154ce91893e7e2e5bbf739a073c4344996af3b72ac6ed0841c9688d7"}
Nov 01 01:03:38.548160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.548120    4493 generic.go:356] "Generic (PLEG): container finished" podID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" containerID="d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33" exitCode=0
Nov 01 01:03:38.548295 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.548234    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"
Nov 01 01:03:38.548295 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.548252    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" event={"ID":"6f3100ef-68f3-4199-9aaf-e4ea15262a55","Type":"ContainerDied","Data":"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"}
Nov 01 01:03:38.548295 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.548320    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp" event={"ID":"6f3100ef-68f3-4199-9aaf-e4ea15262a55","Type":"ContainerDied","Data":"a78ab8c38c1c2fadb1bb93f8784b224efb0aeff5abde1a89c9f376b4e7c683b9"}
Nov 01 01:03:38.556595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.556579    4493 scope.go:122] "RemoveContainer" containerID="29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"
Nov 01 01:03:38.565910 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.565894    4493 scope.go:122] "RemoveContainer" containerID="35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"
Nov 01 01:03:38.566217 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:38.566192    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd\": container with ID starting with 35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd not found: ID does not exist" containerID="35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"
Nov 01 01:03:38.566308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.566219    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd"} err="failed to get container status \"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd\": rpc error: code = NotFound desc = could not find container \"35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd\": container with ID starting with 35290b98964ef7134740da477ec4684b622cfb1ba67d52e0669f950ee318ccfd not found: ID does not exist"
Nov 01 01:03:38.566308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.566238    4493 scope.go:122] "RemoveContainer" containerID="29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"
Nov 01 01:03:38.566471 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:38.566456    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e\": container with ID starting with 29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e not found: ID does not exist" containerID="29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"
Nov 01 01:03:38.566510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.566476    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e"} err="failed to get container status \"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e\": rpc error: code = NotFound desc = could not find container \"29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e\": container with ID starting with 29e1830328422ec53ebfd51cec4108f7d7d275066011191e99ba86bd1435194e not found: ID does not exist"
Nov 01 01:03:38.566510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.566489    4493 scope.go:122] "RemoveContainer" containerID="cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"
Nov 01 01:03:38.574987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.574965    4493 scope.go:122] "RemoveContainer" containerID="cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"
Nov 01 01:03:38.575227 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:38.575203    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358\": container with ID starting with cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358 not found: ID does not exist" containerID="cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"
Nov 01 01:03:38.575308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.575234    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358"} err="failed to get container status \"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358\": rpc error: code = NotFound desc = could not find container \"cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358\": container with ID starting with cc25bda57510fcbcd9f61d3f590770989450e6c336adeac8b7c967c6a3198358 not found: ID does not exist"
Nov 01 01:03:38.575308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.575252    4493 scope.go:122] "RemoveContainer" containerID="d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"
Nov 01 01:03:38.586237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.586217    4493 scope.go:122] "RemoveContainer" containerID="d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"
Nov 01 01:03:38.586549 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:38.586532    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33\": container with ID starting with d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33 not found: ID does not exist" containerID="d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"
Nov 01 01:03:38.586605 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.586556    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33"} err="failed to get container status \"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33\": rpc error: code = NotFound desc = could not find container \"d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33\": container with ID starting with d536a2852cfd1d3ff8da8be08679abee433ac65f018685913eff44fe28765f33 not found: ID does not exist"
Nov 01 01:03:38.769277 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.769196    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 01:03:38.773853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.773827    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e296f9-85997cc7f5-lcdw9"]
Nov 01 01:03:38.872833 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.872794    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 01:03:38.876921 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.876893    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e296f9-58569d744b-2w2dp"]
Nov 01 01:03:38.986455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.986412    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 01:03:38.990654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:38.990626    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af82f1-openshift-default-668b769495-gd2qr"]
Nov 01 01:03:40.503950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:40.503905    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19909aa7-543e-480a-acd2-84655e8abb4d" path="/var/lib/kubelet/pods/19909aa7-543e-480a-acd2-84655e8abb4d/volumes"
Nov 01 01:03:40.504616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:40.504592    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" path="/var/lib/kubelet/pods/3320394e-5d9c-43a9-a2f2-c847ed77f3c5/volumes"
Nov 01 01:03:40.505014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:40.504995    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" path="/var/lib/kubelet/pods/6f3100ef-68f3-4199-9aaf-e4ea15262a55/volumes"
Nov 01 01:03:44.990167 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:44.990106    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:03:45.581267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:45.581228    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 01:03:50.546575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:50.546544    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:03:50.547014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:50.546727    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerName="maas-api" containerID="cri-o://c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a" gracePeriod=30
Nov 01 01:03:50.560524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:50.560489    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.125:8443/health\": dial tcp 10.133.0.125:8443: connect: connection refused"
Nov 01 01:03:50.581601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:50.581546    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" podUID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" containerName="payload-pre-processing" containerID="cri-o://0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd" gracePeriod=25
Nov 01 01:03:50.935627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:50.935597    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:03:51.015386 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.015350    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\" (UniqueName: \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\") pod \"811fabb5-5e40-4002-bed0-2fab51c7d065\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") "
Nov 01 01:03:51.015585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.015415    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") pod \"811fabb5-5e40-4002-bed0-2fab51c7d065\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") "
Nov 01 01:03:51.015585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.015458    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\") pod \"811fabb5-5e40-4002-bed0-2fab51c7d065\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") "
Nov 01 01:03:51.015703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.015617    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\") pod \"811fabb5-5e40-4002-bed0-2fab51c7d065\" (UID: \"811fabb5-5e40-4002-bed0-2fab51c7d065\") "
Nov 01 01:03:51.022482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.022434    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm" pod "811fabb5-5e40-4002-bed0-2fab51c7d065" (UID: "811fabb5-5e40-4002-bed0-2fab51c7d065"). InnerVolumeSpecName "kube-api-access-kngdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:51.023605 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.022890    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs" pod "811fabb5-5e40-4002-bed0-2fab51c7d065" (UID: "811fabb5-5e40-4002-bed0-2fab51c7d065"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:03:51.026493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.026464    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca" pod "811fabb5-5e40-4002-bed0-2fab51c7d065" (UID: "811fabb5-5e40-4002-bed0-2fab51c7d065"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:51.027749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.027726    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 01:03:51.028201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.028122    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls" pod "811fabb5-5e40-4002-bed0-2fab51c7d065" (UID: "811fabb5-5e40-4002-bed0-2fab51c7d065"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:03:51.116948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.116861    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\") pod \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") "
Nov 01 01:03:51.116948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.116904    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\" (UniqueName: \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\") pod \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\" (UID: \"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0\") "
Nov 01 01:03:51.118838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.117553    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngdm\" (UniqueName: \"kubernetes.io/projected/811fabb5-5e40-4002-bed0-2fab51c7d065-kube-api-access-kngdm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.118838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.117577    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.118838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.117588    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811fabb5-5e40-4002-bed0-2fab51c7d065-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.118838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.117597    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811fabb5-5e40-4002-bed0-2fab51c7d065-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.118838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.118249    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume" pod "c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" (UID: "c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:03:51.120597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.120563    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r" pod "c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" (UID: "c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0"). InnerVolumeSpecName "kube-api-access-d652r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:03:51.218442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.218411    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.218442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.218438    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d652r\" (UniqueName: \"kubernetes.io/projected/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0-kube-api-access-d652r\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:03:51.644155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.640310    4493 generic.go:356] "Generic (PLEG): container finished" podID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerID="c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a" exitCode=0
Nov 01 01:03:51.644155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.640439    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" event={"ID":"811fabb5-5e40-4002-bed0-2fab51c7d065","Type":"ContainerDied","Data":"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"}
Nov 01 01:03:51.644155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.640504    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk" event={"ID":"811fabb5-5e40-4002-bed0-2fab51c7d065","Type":"ContainerDied","Data":"155c4e8a4e6dc9b4b46ce3ef243f30924d79d17d04553d9592d6cd51aeef3a80"}
Nov 01 01:03:51.644155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.640532    4493 scope.go:122] "RemoveContainer" containerID="c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"
Nov 01 01:03:51.644155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.640688    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"
Nov 01 01:03:51.658652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.658621    4493 generic.go:356] "Generic (PLEG): container finished" podID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" containerID="0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd" exitCode=0
Nov 01 01:03:51.658814 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.658767    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" event={"ID":"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0","Type":"ContainerDied","Data":"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"}
Nov 01 01:03:51.658876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.658819    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4" event={"ID":"c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0","Type":"ContainerDied","Data":"98a438a650f148b5a4406de676267f7a37797220ed78cdb798a4cee2dd114e14"}
Nov 01 01:03:51.658929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.658891    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"
Nov 01 01:03:51.674548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.672069    4493 scope.go:122] "RemoveContainer" containerID="c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"
Nov 01 01:03:51.674548 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:51.672719    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a\": container with ID starting with c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a not found: ID does not exist" containerID="c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"
Nov 01 01:03:51.674548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.672754    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a"} err="failed to get container status \"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a\": rpc error: code = NotFound desc = could not find container \"c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a\": container with ID starting with c612c60a668d946177718e0028677fa1482fe9f734a8fbdbc463f28b16fdf54a not found: ID does not exist"
Nov 01 01:03:51.674548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.672781    4493 scope.go:122] "RemoveContainer" containerID="0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"
Nov 01 01:03:51.690992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.690967    4493 scope.go:122] "RemoveContainer" containerID="0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"
Nov 01 01:03:51.691404 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:03:51.691359    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd\": container with ID starting with 0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd not found: ID does not exist" containerID="0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"
Nov 01 01:03:51.691518 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.691414    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd"} err="failed to get container status \"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd\": rpc error: code = NotFound desc = could not find container \"0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd\": container with ID starting with 0cd2ca48619cca10c992c91a9ba53de79fa3290950a0e297c350d6fa56916ecd not found: ID does not exist"
Nov 01 01:03:51.815664 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.815605    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 01:03:51.819320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.819295    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc241b-6f7f6b958c-pv4z4"]
Nov 01 01:03:51.928342 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.928266    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:03:51.934545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:51.934518    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bc241b-7b4c559f5f-wgnnk"]
Nov 01 01:03:52.503791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:52.503758    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" path="/var/lib/kubelet/pods/811fabb5-5e40-4002-bed0-2fab51c7d065/volumes"
Nov 01 01:03:52.504196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:03:52.504183    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" path="/var/lib/kubelet/pods/c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0/volumes"
Nov 01 01:04:14.989779 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:14.989743    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:04:15.345898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:15.345815    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 01:04:15.458943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:15.458906    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 01:04:16.858167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.856241    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 01:04:16.858167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.856491    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" containerID="cri-o://6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f" gracePeriod=30
Nov 01 01:04:16.901266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.901229    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 01:04:16.901731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.901700    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy" containerID="cri-o://daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe" gracePeriod=30
Nov 01 01:04:16.915891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.902530    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 01:04:16.915891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.902708    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" containerID="cri-o://57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385" gracePeriod=30
Nov 01 01:04:16.939849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.939813    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 01:04:16.940114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:16.940069    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy" containerID="cri-o://f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24" gracePeriod=30
Nov 01 01:04:18.484591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:18.484555    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qfr9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:04:18.485072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:18.484622    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:04:18.489496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:18.489468    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rswbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:04:18.489625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:18.489521    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:04:20.347090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.347050    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" podUID="3da4368c-309d-4db3-8f66-353a68198ee9" containerName="payload-processing" containerID="cri-o://1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4" gracePeriod=25
Nov 01 01:04:20.460235 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.460179    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" podUID="21233329-fa85-43ee-a4e9-9f41ff989632" containerName="payload-pre-processing" containerID="cri-o://72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc" gracePeriod=25
Nov 01 01:04:20.618912 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.618852    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 01:04:20.716955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.716927    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\" (UniqueName: \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\") pod \"3da4368c-309d-4db3-8f66-353a68198ee9\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") "
Nov 01 01:04:20.717127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.717077    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\") pod \"3da4368c-309d-4db3-8f66-353a68198ee9\" (UID: \"3da4368c-309d-4db3-8f66-353a68198ee9\") "
Nov 01 01:04:20.717449 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.717424    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume" pod "3da4368c-309d-4db3-8f66-353a68198ee9" (UID: "3da4368c-309d-4db3-8f66-353a68198ee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:20.717805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.717697    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da4368c-309d-4db3-8f66-353a68198ee9-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:20.719384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.719347    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z" pod "3da4368c-309d-4db3-8f66-353a68198ee9" (UID: "3da4368c-309d-4db3-8f66-353a68198ee9"). InnerVolumeSpecName "kube-api-access-g7d7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:20.739884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.739863    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 01:04:20.818625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.818594    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\") pod \"21233329-fa85-43ee-a4e9-9f41ff989632\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") "
Nov 01 01:04:20.818814 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.818646    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\" (UniqueName: \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\") pod \"21233329-fa85-43ee-a4e9-9f41ff989632\" (UID: \"21233329-fa85-43ee-a4e9-9f41ff989632\") "
Nov 01 01:04:20.819030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.819007    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume" pod "21233329-fa85-43ee-a4e9-9f41ff989632" (UID: "21233329-fa85-43ee-a4e9-9f41ff989632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:20.819104 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.819052    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7d7z\" (UniqueName: \"kubernetes.io/projected/3da4368c-309d-4db3-8f66-353a68198ee9-kube-api-access-g7d7z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:20.820911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.820894    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x" pod "21233329-fa85-43ee-a4e9-9f41ff989632" (UID: "21233329-fa85-43ee-a4e9-9f41ff989632"). InnerVolumeSpecName "kube-api-access-nnf6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:20.920050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.919974    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnf6x\" (UniqueName: \"kubernetes.io/projected/21233329-fa85-43ee-a4e9-9f41ff989632-kube-api-access-nnf6x\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:20.920050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.920001    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21233329-fa85-43ee-a4e9-9f41ff989632-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:20.958927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.958896    4493 generic.go:356] "Generic (PLEG): container finished" podID="21233329-fa85-43ee-a4e9-9f41ff989632" containerID="72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc" exitCode=0
Nov 01 01:04:20.959103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.958964    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"
Nov 01 01:04:20.959103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.958989    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" event={"ID":"21233329-fa85-43ee-a4e9-9f41ff989632","Type":"ContainerDied","Data":"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"}
Nov 01 01:04:20.959103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.959044    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh" event={"ID":"21233329-fa85-43ee-a4e9-9f41ff989632","Type":"ContainerDied","Data":"d4480752c7312b2242a70a43b2f644ec00e35b1e8bc08c86a869038fb3b013fb"}
Nov 01 01:04:20.959103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.959075    4493 scope.go:122] "RemoveContainer" containerID="72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"
Nov 01 01:04:20.960628 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.960605    4493 generic.go:356] "Generic (PLEG): container finished" podID="3da4368c-309d-4db3-8f66-353a68198ee9" containerID="1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4" exitCode=0
Nov 01 01:04:20.960746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.960660    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"
Nov 01 01:04:20.960746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.960677    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" event={"ID":"3da4368c-309d-4db3-8f66-353a68198ee9","Type":"ContainerDied","Data":"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"}
Nov 01 01:04:20.960746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.960716    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf" event={"ID":"3da4368c-309d-4db3-8f66-353a68198ee9","Type":"ContainerDied","Data":"11c39f211b155a3ef0125ec190a51e1bf58f0345bb1e6ac5ddd0f9b5b61917ff"}
Nov 01 01:04:20.970684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.970669    4493 scope.go:122] "RemoveContainer" containerID="72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"
Nov 01 01:04:20.970968 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:20.970951    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc\": container with ID starting with 72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc not found: ID does not exist" containerID="72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"
Nov 01 01:04:20.971019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.970975    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc"} err="failed to get container status \"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc\": rpc error: code = NotFound desc = could not find container \"72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc\": container with ID starting with 72830ca10cb453ccc1a143f4d9efc6cd6c9a4928022f935432bced4344af94dc not found: ID does not exist"
Nov 01 01:04:20.971019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.970993    4493 scope.go:122] "RemoveContainer" containerID="1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"
Nov 01 01:04:20.979824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.979806    4493 scope.go:122] "RemoveContainer" containerID="1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"
Nov 01 01:04:20.980142 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:20.980078    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4\": container with ID starting with 1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4 not found: ID does not exist" containerID="1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"
Nov 01 01:04:20.980218 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:20.980153    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4"} err="failed to get container status \"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4\": rpc error: code = NotFound desc = could not find container \"1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4\": container with ID starting with 1a96a958b045bd1afe42e8cb2d195f611bf00c1611ba2ea6febcdb95cf3875f4 not found: ID does not exist"
Nov 01 01:04:21.163604 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:21.163557    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 01:04:21.167222 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:21.167199    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f97e89-7bf59f4654-lh8gf"]
Nov 01 01:04:21.276221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:21.276187    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 01:04:21.280698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:21.280671    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f97e89-6fd86d6f99-hxwbh"]
Nov 01 01:04:22.161090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.161057    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 01:04:22.219467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.219441    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 01:04:22.237177 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237142    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237211    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237296    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237322    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237355    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237417    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237446    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237511    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237559    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237594    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237638    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237664    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237713    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.237797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237734    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.238101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237802    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.238101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237874    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\") pod \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\" (UID: \"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421\") "
Nov 01 01:04:22.238101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237903    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.238101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.237964    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\") pod \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\" (UID: \"c5ab1e26-cf86-4ada-9e27-848ef59d1a81\") "
Nov 01 01:04:22.238590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.238562    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.239081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239058    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.239342 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239316    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.239445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239419    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.239589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239564    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.239652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239639    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.239652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239643    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.240078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239881    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.240078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.239964    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:22.240078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.240006    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.240648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.240622    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:22.242560 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.242522    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.243873 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.243851    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:04:22.249704 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.249656    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "kube-api-access-rs645". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.249806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.249734    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "kube-api-access-slzdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.249806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.249740    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.249806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.249764    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.249975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.249854    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token" pod "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" (UID: "7ed7b9c2-8011-414b-8bc4-f8acd2bf7421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.250869 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.250844    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo" pod "c5ab1e26-cf86-4ada-9e27-848ef59d1a81" (UID: "c5ab1e26-cf86-4ada-9e27-848ef59d1a81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:04:22.275294 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.275273    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 01:04:22.293898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.293876    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 01:04:22.342568 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.342533    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.342838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.342813    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.342941 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.342905    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343002 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.342956    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343002    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343032    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343064    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343088    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:22.343202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343116    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343236    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343267    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343314    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343380    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.343533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343439    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343470    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343513    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343556    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.343775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343578    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.343775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343598    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\") pod \"3e5f909a-afc3-4541-9c91-2d692f814a37\" (UID: \"3e5f909a-afc3-4541-9c91-2d692f814a37\") "
Nov 01 01:04:22.343775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.343642    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\") pod \"2c7df77c-4996-41de-b23d-020634c40e77\" (UID: \"2c7df77c-4996-41de-b23d-020634c40e77\") "
Nov 01 01:04:22.344196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344173    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344201    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344216    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344229    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344243    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344257    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344272    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs645\" (UniqueName: \"kubernetes.io/projected/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-kube-api-access-rs645\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344287    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344301    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344314    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344328    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344342    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344356    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzdt\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-kube-api-access-slzdt\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344371    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344386    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344400    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344413    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7df77c-4996-41de-b23d-020634c40e77-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344427    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344441    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.344657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344467    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1e26-cf86-4ada-9e27-848ef59d1a81-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.345289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.344703    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.345289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.345280    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.345930 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.345496    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.345930 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.345721    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.346091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.346068    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:22.346674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.346398    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.347036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.346960    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.347180 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.347154    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.350343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350225    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:04:22.350343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350235    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:04:22.350343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350284    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "kube-api-access-rbsv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.350343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350324    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.350517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350410    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token" pod "2c7df77c-4996-41de-b23d-020634c40e77" (UID: "2c7df77c-4996-41de-b23d-020634c40e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.350733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.350711    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "kube-api-access-5jvwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:22.351610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.351592    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy" pod "3e5f909a-afc3-4541-9c91-2d692f814a37" (UID: "3e5f909a-afc3-4541-9c91-2d692f814a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445680    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445713    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445727    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbsv8\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-kube-api-access-rbsv8\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445744    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445755    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445768    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.445786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445780    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445795    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7df77c-4996-41de-b23d-020634c40e77-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445808    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5f909a-afc3-4541-9c91-2d692f814a37-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445821    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7df77c-4996-41de-b23d-020634c40e77-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445832    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445844    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445854    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5f909a-afc3-4541-9c91-2d692f814a37-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445866    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7df77c-4996-41de-b23d-020634c40e77-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.446244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.445878    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvwx\" (UniqueName: \"kubernetes.io/projected/3e5f909a-afc3-4541-9c91-2d692f814a37-kube-api-access-5jvwx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:22.504040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.504007    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21233329-fa85-43ee-a4e9-9f41ff989632" path="/var/lib/kubelet/pods/21233329-fa85-43ee-a4e9-9f41ff989632/volumes"
Nov 01 01:04:22.504496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.504481    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da4368c-309d-4db3-8f66-353a68198ee9" path="/var/lib/kubelet/pods/3da4368c-309d-4db3-8f66-353a68198ee9/volumes"
Nov 01 01:04:22.975932 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.975899    4493 generic.go:356] "Generic (PLEG): container finished" podID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerID="57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385" exitCode=0
Nov 01 01:04:22.976205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.975959    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"
Nov 01 01:04:22.976205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.975984    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" event={"ID":"c5ab1e26-cf86-4ada-9e27-848ef59d1a81","Type":"ContainerDied","Data":"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"}
Nov 01 01:04:22.976205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.976043    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl" event={"ID":"c5ab1e26-cf86-4ada-9e27-848ef59d1a81","Type":"ContainerDied","Data":"e30c849c148876432337e236f4fe2966d94d590ac9795a5b84295532fcbe4461"}
Nov 01 01:04:22.976205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.976076    4493 scope.go:122] "RemoveContainer" containerID="57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"
Nov 01 01:04:22.977517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.977488    4493 generic.go:356] "Generic (PLEG): container finished" podID="2c7df77c-4996-41de-b23d-020634c40e77" containerID="f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24" exitCode=0
Nov 01 01:04:22.977644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.977555    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"
Nov 01 01:04:22.977644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.977581    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" event={"ID":"2c7df77c-4996-41de-b23d-020634c40e77","Type":"ContainerDied","Data":"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"}
Nov 01 01:04:22.977644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.977623    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs" event={"ID":"2c7df77c-4996-41de-b23d-020634c40e77","Type":"ContainerDied","Data":"cb14b66b2887d85bc7158ac0112fa86fdd77b440413b88092aed5b3847e4f3c4"}
Nov 01 01:04:22.979378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.979348    4493 generic.go:356] "Generic (PLEG): container finished" podID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerID="daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe" exitCode=0
Nov 01 01:04:22.979519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.979385    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" event={"ID":"3e5f909a-afc3-4541-9c91-2d692f814a37","Type":"ContainerDied","Data":"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"}
Nov 01 01:04:22.979519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.979427    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"
Nov 01 01:04:22.979519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.979452    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88" event={"ID":"3e5f909a-afc3-4541-9c91-2d692f814a37","Type":"ContainerDied","Data":"bb795282843ce483019af27fc35df58df313e3cf2e04d042433431cb8e8cb78f"}
Nov 01 01:04:22.981115 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.981095    4493 generic.go:356] "Generic (PLEG): container finished" podID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerID="6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f" exitCode=0
Nov 01 01:04:22.981262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.981159    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" event={"ID":"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421","Type":"ContainerDied","Data":"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"}
Nov 01 01:04:22.981262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.981189    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"
Nov 01 01:04:22.981262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.981197    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d" event={"ID":"7ed7b9c2-8011-414b-8bc4-f8acd2bf7421","Type":"ContainerDied","Data":"325598a3d044da0363f78557254998be10c754a6c3fe1c295f686e639f1b776f"}
Nov 01 01:04:22.987695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.987521    4493 scope.go:122] "RemoveContainer" containerID="57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"
Nov 01 01:04:22.987821 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:22.987804    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385\": container with ID starting with 57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385 not found: ID does not exist" containerID="57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"
Nov 01 01:04:22.987866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.987829    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385"} err="failed to get container status \"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385\": rpc error: code = NotFound desc = could not find container \"57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385\": container with ID starting with 57f4b2856d8d5c8df14201edb31f1182218383eb08286daff9df96fcdfe7e385 not found: ID does not exist"
Nov 01 01:04:22.987866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.987846    4493 scope.go:122] "RemoveContainer" containerID="f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"
Nov 01 01:04:22.999632 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.999609    4493 scope.go:122] "RemoveContainer" containerID="f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"
Nov 01 01:04:22.999889 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:22.999869    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24\": container with ID starting with f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24 not found: ID does not exist" containerID="f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"
Nov 01 01:04:22.999934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.999896    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24"} err="failed to get container status \"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24\": rpc error: code = NotFound desc = could not find container \"f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24\": container with ID starting with f16d6845c44e436092e4c01f9e68b26df4bb3d2d1f2e57858cfb908bf66e0e24 not found: ID does not exist"
Nov 01 01:04:22.999934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:22.999913    4493 scope.go:122] "RemoveContainer" containerID="daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"
Nov 01 01:04:23.009024 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.009008    4493 scope.go:122] "RemoveContainer" containerID="daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"
Nov 01 01:04:23.009319 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:23.009302    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe\": container with ID starting with daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe not found: ID does not exist" containerID="daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"
Nov 01 01:04:23.009384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.009326    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe"} err="failed to get container status \"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe\": rpc error: code = NotFound desc = could not find container \"daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe\": container with ID starting with daf4ecc074183388385985b37cdc74dd9431b4d222a737afb5ba2462ddb720fe not found: ID does not exist"
Nov 01 01:04:23.009384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.009343    4493 scope.go:122] "RemoveContainer" containerID="6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"
Nov 01 01:04:23.017895 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.017880    4493 scope.go:122] "RemoveContainer" containerID="6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"
Nov 01 01:04:23.018177 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:23.018155    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f\": container with ID starting with 6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f not found: ID does not exist" containerID="6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"
Nov 01 01:04:23.018261 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.018183    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f"} err="failed to get container status \"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f\": rpc error: code = NotFound desc = could not find container \"6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f\": container with ID starting with 6d82540f7bd22fb04894fc2a985c74d7a2a610a2dd5e258b833ffcfad9ffa02f not found: ID does not exist"
Nov 01 01:04:23.142625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.142597    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 01:04:23.146495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.146467    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qfr9d"]
Nov 01 01:04:23.257461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.257382    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 01:04:23.263044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.263017    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rvm88"]
Nov 01 01:04:23.373385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.373351    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 01:04:23.378069 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.378043    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rswbl"]
Nov 01 01:04:23.487740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.487710    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 01:04:23.493276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:23.493249    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qplbs"]
Nov 01 01:04:24.502798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:24.502769    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7df77c-4996-41de-b23d-020634c40e77" path="/var/lib/kubelet/pods/2c7df77c-4996-41de-b23d-020634c40e77/volumes"
Nov 01 01:04:24.503267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:24.503252    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" path="/var/lib/kubelet/pods/3e5f909a-afc3-4541-9c91-2d692f814a37/volumes"
Nov 01 01:04:24.503718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:24.503706    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" path="/var/lib/kubelet/pods/7ed7b9c2-8011-414b-8bc4-f8acd2bf7421/volumes"
Nov 01 01:04:24.504470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:24.504454    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" path="/var/lib/kubelet/pods/c5ab1e26-cf86-4ada-9e27-848ef59d1a81/volumes"
Nov 01 01:04:34.158019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:34.157984    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:04:42.753192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:42.753107    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 01:04:42.753593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:42.753295    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" containerID="cri-o://7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3" gracePeriod=30
Nov 01 01:04:43.107953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:43.107859    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:04:43.107953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:43.107935    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:04:44.990178 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:44.990115    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:04:45.654168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:45.654120    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 01:04:45.654367 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:45.654345    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" podUID="e878bb21-6c71-4b49-9b17-8c3297deca21" containerName="maas-api" containerID="cri-o://5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf" gracePeriod=30
Nov 01 01:04:45.939825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:45.939797    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 01:04:46.123319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.120915    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\") pod \"e878bb21-6c71-4b49-9b17-8c3297deca21\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") "
Nov 01 01:04:46.123319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.121020    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\") pod \"e878bb21-6c71-4b49-9b17-8c3297deca21\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") "
Nov 01 01:04:46.123319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.121103    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\" (UniqueName: \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\") pod \"e878bb21-6c71-4b49-9b17-8c3297deca21\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") "
Nov 01 01:04:46.123319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.121152    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") pod \"e878bb21-6c71-4b49-9b17-8c3297deca21\" (UID: \"e878bb21-6c71-4b49-9b17-8c3297deca21\") "
Nov 01 01:04:46.123319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.122071    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca" pod "e878bb21-6c71-4b49-9b17-8c3297deca21" (UID: "e878bb21-6c71-4b49-9b17-8c3297deca21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:46.124010 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.123880    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs" pod "e878bb21-6c71-4b49-9b17-8c3297deca21" (UID: "e878bb21-6c71-4b49-9b17-8c3297deca21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:46.126407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.124968    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx" pod "e878bb21-6c71-4b49-9b17-8c3297deca21" (UID: "e878bb21-6c71-4b49-9b17-8c3297deca21"). InnerVolumeSpecName "kube-api-access-6vwfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:46.126407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.125244    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls" pod "e878bb21-6c71-4b49-9b17-8c3297deca21" (UID: "e878bb21-6c71-4b49-9b17-8c3297deca21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:46.128690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.128662    4493 generic.go:356] "Generic (PLEG): container finished" podID="e878bb21-6c71-4b49-9b17-8c3297deca21" containerID="5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf" exitCode=0
Nov 01 01:04:46.128834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.128709    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"
Nov 01 01:04:46.128834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.128740    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" event={"ID":"e878bb21-6c71-4b49-9b17-8c3297deca21","Type":"ContainerDied","Data":"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"}
Nov 01 01:04:46.128834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.128792    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7" event={"ID":"e878bb21-6c71-4b49-9b17-8c3297deca21","Type":"ContainerDied","Data":"905dce0e3d695e4a47e6be37b4a609aecb27bf7a8d847f6ddbae1aab349a9fa7"}
Nov 01 01:04:46.128834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.128816    4493 scope.go:122] "RemoveContainer" containerID="5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"
Nov 01 01:04:46.151667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.151647    4493 scope.go:122] "RemoveContainer" containerID="5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"
Nov 01 01:04:46.151986 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:46.151965    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf\": container with ID starting with 5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf not found: ID does not exist" containerID="5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"
Nov 01 01:04:46.152043 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.151993    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf"} err="failed to get container status \"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf\": rpc error: code = NotFound desc = could not find container \"5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf\": container with ID starting with 5897f69e5eebd6ff5963227ad0fbca3522d2563801b0f09eb44974d65e1d99cf not found: ID does not exist"
Nov 01 01:04:46.222982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.222949    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e878bb21-6c71-4b49-9b17-8c3297deca21-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:46.222982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.222981    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwfx\" (UniqueName: \"kubernetes.io/projected/e878bb21-6c71-4b49-9b17-8c3297deca21-kube-api-access-6vwfx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:46.223251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.223002    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:46.223251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.223011    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e878bb21-6c71-4b49-9b17-8c3297deca21-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:46.295754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.295720    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 01:04:46.299776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.299752    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ba5b10-56dbc94f45-ntjm7"]
Nov 01 01:04:46.502945 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.502867    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e878bb21-6c71-4b49-9b17-8c3297deca21" path="/var/lib/kubelet/pods/e878bb21-6c71-4b49-9b17-8c3297deca21/volumes"
Nov 01 01:04:46.838856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.838781    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:04:46.839038 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:46.838979    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" podUID="13079830-5111-4d67-873a-812bd9038a5e" containerName="maas-api" containerID="cri-o://752f036d7b5f7ee0592ac0d3f40e01c6669a190af7c37e03485dc5dd703e5b21" gracePeriod=30
Nov 01 01:04:47.139539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.139307    4493 generic.go:356] "Generic (PLEG): container finished" podID="13079830-5111-4d67-873a-812bd9038a5e" containerID="752f036d7b5f7ee0592ac0d3f40e01c6669a190af7c37e03485dc5dd703e5b21" exitCode=0
Nov 01 01:04:47.139539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.139514    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" event={"ID":"13079830-5111-4d67-873a-812bd9038a5e","Type":"ContainerDied","Data":"752f036d7b5f7ee0592ac0d3f40e01c6669a190af7c37e03485dc5dd703e5b21"}
Nov 01 01:04:47.164770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.164746    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:04:47.348391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.348343    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") pod \"13079830-5111-4d67-873a-812bd9038a5e\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") "
Nov 01 01:04:47.348562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.348436    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\") pod \"13079830-5111-4d67-873a-812bd9038a5e\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") "
Nov 01 01:04:47.348562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.348485    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\") pod \"13079830-5111-4d67-873a-812bd9038a5e\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") "
Nov 01 01:04:47.348769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.348686    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\" (UniqueName: \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\") pod \"13079830-5111-4d67-873a-812bd9038a5e\" (UID: \"13079830-5111-4d67-873a-812bd9038a5e\") "
Nov 01 01:04:47.348942 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.348916    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca" pod "13079830-5111-4d67-873a-812bd9038a5e" (UID: "13079830-5111-4d67-873a-812bd9038a5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:47.349472 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.349446    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13079830-5111-4d67-873a-812bd9038a5e-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:47.350773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.350744    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls" pod "13079830-5111-4d67-873a-812bd9038a5e" (UID: "13079830-5111-4d67-873a-812bd9038a5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:47.350909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.350890    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs" pod "13079830-5111-4d67-873a-812bd9038a5e" (UID: "13079830-5111-4d67-873a-812bd9038a5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:47.351105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.351082    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2" pod "13079830-5111-4d67-873a-812bd9038a5e" (UID: "13079830-5111-4d67-873a-812bd9038a5e"). InnerVolumeSpecName "kube-api-access-rccj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:47.450955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.450873    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccj2\" (UniqueName: \"kubernetes.io/projected/13079830-5111-4d67-873a-812bd9038a5e-kube-api-access-rccj2\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:47.450955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.450913    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:47.450955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:47.450927    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13079830-5111-4d67-873a-812bd9038a5e-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.049765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.049741    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 01:04:48.148119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.148094    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"
Nov 01 01:04:48.148576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.148157    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46" event={"ID":"13079830-5111-4d67-873a-812bd9038a5e","Type":"ContainerDied","Data":"83f09676c3d45d1f32bd537150615a0f0696af8f414302d4c958b9e1aacb427b"}
Nov 01 01:04:48.148576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.148215    4493 scope.go:122] "RemoveContainer" containerID="752f036d7b5f7ee0592ac0d3f40e01c6669a190af7c37e03485dc5dd703e5b21"
Nov 01 01:04:48.149710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.149690    4493 generic.go:356] "Generic (PLEG): container finished" podID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerID="7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3" exitCode=0
Nov 01 01:04:48.149927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.149727    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" event={"ID":"7d6cc26a-190f-4eb2-8146-0069933a0456","Type":"ContainerDied","Data":"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"}
Nov 01 01:04:48.149927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.149746    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"
Nov 01 01:04:48.149927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.149772    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf" event={"ID":"7d6cc26a-190f-4eb2-8146-0069933a0456","Type":"ContainerDied","Data":"e0873a010f84ad60ee977bd772ecf6186893b6db431ea716d3554ab7d0ab5a7c"}
Nov 01 01:04:48.158735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.158718    4493 scope.go:122] "RemoveContainer" containerID="7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"
Nov 01 01:04:48.168194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.168176    4493 scope.go:122] "RemoveContainer" containerID="7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"
Nov 01 01:04:48.168458 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:48.168438    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3\": container with ID starting with 7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3 not found: ID does not exist" containerID="7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"
Nov 01 01:04:48.168495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.168468    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3"} err="failed to get container status \"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3\": rpc error: code = NotFound desc = could not find container \"7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3\": container with ID starting with 7001d7c3259771da082199df8b3e16ded91472f05cc233a0d382de20017234d3 not found: ID does not exist"
Nov 01 01:04:48.170825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.170808    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.170875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.170868    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.170928 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.170901    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.170987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.170968    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171026    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171042    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171072    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171180    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171226    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\") pod \"7d6cc26a-190f-4eb2-8146-0069933a0456\" (UID: \"7d6cc26a-190f-4eb2-8146-0069933a0456\") "
Nov 01 01:04:48.171342 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171323    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:48.171530 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171414    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:48.171530 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171529    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:48.171798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171701    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:48.171861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171848    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6cc26a-190f-4eb2-8146-0069933a0456-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.171861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171848    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:48.171975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171864    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.171975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171875    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.171975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.171884    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.173657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.173636    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:04:48.173657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.173640    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "kube-api-access-j7b8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:48.182463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.182442    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:04:48.193477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.193453    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token" pod "7d6cc26a-190f-4eb2-8146-0069933a0456" (UID: "7d6cc26a-190f-4eb2-8146-0069933a0456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:48.272856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.272816    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.272856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.272842    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.272856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.272851    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.272856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.272860    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6cc26a-190f-4eb2-8146-0069933a0456-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.272856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.272869    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7b8w\" (UniqueName: \"kubernetes.io/projected/7d6cc26a-190f-4eb2-8146-0069933a0456-kube-api-access-j7b8w\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:48.384879 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.384790    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:04:48.391280 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.391249    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f9d81-6664f6bbdd-w4d46"]
Nov 01 01:04:48.502813 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.502780    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13079830-5111-4d67-873a-812bd9038a5e" path="/var/lib/kubelet/pods/13079830-5111-4d67-873a-812bd9038a5e/volumes"
Nov 01 01:04:48.579789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.579760    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 01:04:48.583556 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:48.583528    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-2ngpf"]
Nov 01 01:04:50.503243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:50.503213    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" path="/var/lib/kubelet/pods/7d6cc26a-190f-4eb2-8146-0069933a0456/volumes"
Nov 01 01:04:53.474206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.474171    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:04:53.474798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.474402    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" podUID="42933fe6-2288-4863-8602-c47e638d1619" containerName="maas-api" containerID="cri-o://745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c" gracePeriod=30
Nov 01 01:04:53.814513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.814489    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:04:53.843595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.843565    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\") pod \"42933fe6-2288-4863-8602-c47e638d1619\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") "
Nov 01 01:04:53.843748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.843612    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") pod \"42933fe6-2288-4863-8602-c47e638d1619\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") "
Nov 01 01:04:53.843748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.843690    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\") pod \"42933fe6-2288-4863-8602-c47e638d1619\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") "
Nov 01 01:04:53.843748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.843721    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\" (UniqueName: \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\") pod \"42933fe6-2288-4863-8602-c47e638d1619\" (UID: \"42933fe6-2288-4863-8602-c47e638d1619\") "
Nov 01 01:04:53.844128 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.844098    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca" pod "42933fe6-2288-4863-8602-c47e638d1619" (UID: "42933fe6-2288-4863-8602-c47e638d1619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:04:53.845855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.845834    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs" pod "42933fe6-2288-4863-8602-c47e638d1619" (UID: "42933fe6-2288-4863-8602-c47e638d1619"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:53.845964 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.845872    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls" pod "42933fe6-2288-4863-8602-c47e638d1619" (UID: "42933fe6-2288-4863-8602-c47e638d1619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:04:53.845964 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.845899    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t" pod "42933fe6-2288-4863-8602-c47e638d1619" (UID: "42933fe6-2288-4863-8602-c47e638d1619"). InnerVolumeSpecName "kube-api-access-k5p6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:04:53.945767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.945724    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:53.945767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.945766    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42933fe6-2288-4863-8602-c47e638d1619-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:53.945962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.945787    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42933fe6-2288-4863-8602-c47e638d1619-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:53.945962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:53.945802    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5p6t\" (UniqueName: \"kubernetes.io/projected/42933fe6-2288-4863-8602-c47e638d1619-kube-api-access-k5p6t\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:04:54.185828 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.185752    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"
Nov 01 01:04:54.185828 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.185776    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" event={"ID":"42933fe6-2288-4863-8602-c47e638d1619","Type":"ContainerDied","Data":"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"}
Nov 01 01:04:54.185828 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.185829    4493 scope.go:122] "RemoveContainer" containerID="745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"
Nov 01 01:04:54.186070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.185753    4493 generic.go:356] "Generic (PLEG): container finished" podID="42933fe6-2288-4863-8602-c47e638d1619" containerID="745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c" exitCode=0
Nov 01 01:04:54.186070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.185962    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx" event={"ID":"42933fe6-2288-4863-8602-c47e638d1619","Type":"ContainerDied","Data":"e642878b54c5bd770511bbf7bc8741be8304940ea886c8f8deae360694f02d70"}
Nov 01 01:04:54.195936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.195915    4493 scope.go:122] "RemoveContainer" containerID="745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"
Nov 01 01:04:54.196238 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:04:54.196217    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c\": container with ID starting with 745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c not found: ID does not exist" containerID="745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"
Nov 01 01:04:54.196294 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.196249    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c"} err="failed to get container status \"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c\": rpc error: code = NotFound desc = could not find container \"745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c\": container with ID starting with 745cde4be326d3febf0b2d0db13a1d3ead6b470452ff5019b064bdd045d30c6c not found: ID does not exist"
Nov 01 01:04:54.350986 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.350950    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:04:54.354737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.354714    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e296f9-5594dbdfdf-w2ktx"]
Nov 01 01:04:54.502509 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:04:54.502482    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42933fe6-2288-4863-8602-c47e638d1619" path="/var/lib/kubelet/pods/42933fe6-2288-4863-8602-c47e638d1619/volumes"
Nov 01 01:05:04.902612 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.902577    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:05:04.903113 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903099    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903115    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903148    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903154    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903164    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3da4368c-309d-4db3-8f66-353a68198ee9" containerName="payload-processing"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903170    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da4368c-309d-4db3-8f66-353a68198ee9" containerName="payload-processing"
Nov 01 01:05:04.903179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903179    4493 cpu_manager.go:414] "containerMap: removing container" podUID="42933fe6-2288-4863-8602-c47e638d1619" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903184    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42933fe6-2288-4863-8602-c47e638d1619" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903202    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903206    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903214    4493 cpu_manager.go:414] "containerMap: removing container" podUID="13079830-5111-4d67-873a-812bd9038a5e" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903220    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13079830-5111-4d67-873a-812bd9038a5e" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903235    4493 cpu_manager.go:414] "containerMap: removing container" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903240    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903245    4493 cpu_manager.go:414] "containerMap: removing container" podUID="21233329-fa85-43ee-a4e9-9f41ff989632" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903250    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21233329-fa85-43ee-a4e9-9f41ff989632" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903255    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903259    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903266    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903270    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903276    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903281    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903288    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903296    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903307    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903312    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903318    4493 cpu_manager.go:414] "containerMap: removing container" podUID="19909aa7-543e-480a-acd2-84655e8abb4d" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903322    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19909aa7-543e-480a-acd2-84655e8abb4d" containerName="payload-pre-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903329    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" containerName="payload-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903333    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" containerName="payload-processing"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903339    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e878bb21-6c71-4b49-9b17-8c3297deca21" containerName="maas-api"
Nov 01 01:05:04.903359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903343    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e878bb21-6c71-4b49-9b17-8c3297deca21" containerName="maas-api"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903447    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903460    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed7b9c2-8011-414b-8bc4-f8acd2bf7421" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903469    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da4368c-309d-4db3-8f66-353a68198ee9" containerName="payload-processing"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903475    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="42933fe6-2288-4863-8602-c47e638d1619" containerName="maas-api"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903483    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3320394e-5d9c-43a9-a2f2-c847ed77f3c5" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903488    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="13079830-5111-4d67-873a-812bd9038a5e" containerName="maas-api"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903494    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="811fabb5-5e40-4002-bed0-2fab51c7d065" containerName="maas-api"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903498    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="21233329-fa85-43ee-a4e9-9f41ff989632" containerName="payload-pre-processing"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903504    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6cc26a-190f-4eb2-8146-0069933a0456" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903510    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7df77c-4996-41de-b23d-020634c40e77" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903514    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5f909a-afc3-4541-9c91-2d692f814a37" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903519    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ef3bd9-8823-4b6d-a1ab-6aa508ea32d0" containerName="payload-pre-processing"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903525    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ab1e26-cf86-4ada-9e27-848ef59d1a81" containerName="istio-proxy"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903530    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="19909aa7-543e-480a-acd2-84655e8abb4d" containerName="payload-pre-processing"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903535    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3100ef-68f3-4199-9aaf-e4ea15262a55" containerName="payload-processing"
Nov 01 01:05:04.904093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.903540    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e878bb21-6c71-4b49-9b17-8c3297deca21" containerName="maas-api"
Nov 01 01:05:04.906769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.906754    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.915970    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916117 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916088    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916311 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916229    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916311 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916276    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zgzp\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916377    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916450 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916414    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916529 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916494    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916637 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916617    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.916734 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.916714    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:04.959373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:04.959329    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-718e19-openshift-default-dockercfg-fr2bz\""
Nov 01 01:05:05.013314 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.013279    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:05:05.017656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017626    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zgzp\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.017826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017697    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.017826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017732    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.017826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017772    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017828    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017910    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.017944    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.018019    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018281 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.018050    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018379 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.018351    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.018690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.018654    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.019060 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.019035    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.019382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.019354    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.019558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.019359    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.021385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.021357    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.028743 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.024969    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.106927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.079392    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zgzp\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.106927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.080012    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\") pod \"e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") " pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.220734 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.220695    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:05.657346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.657323    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:05:05.659230 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:05:05.659184    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd055b277_3003_4362_8457_2852717632b3.slice/crio-417d5aba4ff1688c2efc96924ef1f27b6955537aa365a0ab6c7a78df549bfbf2 WatchSource:0}: Error finding container 417d5aba4ff1688c2efc96924ef1f27b6955537aa365a0ab6c7a78df549bfbf2: Status 404 returned error can't find the container with id 417d5aba4ff1688c2efc96924ef1f27b6955537aa365a0ab6c7a78df549bfbf2
Nov 01 01:05:05.661845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.661810    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:05:05.661925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.661894    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:05:05.661966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:05.661941    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:05:06.279192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:06.279152    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerStarted","Data":"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"}
Nov 01 01:05:06.279605 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:06.279209    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerStarted","Data":"417d5aba4ff1688c2efc96924ef1f27b6955537aa365a0ab6c7a78df549bfbf2"}
Nov 01 01:05:06.400936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:06.400874    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podStartSLOduration=2.400860151 podStartE2EDuration="2.400860151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:05:06.398841452 +0000 UTC m=+2496.076428398" watchObservedRunningTime="2026-11-01 01:05:06.400860151 +0000 UTC m=+2496.078447093"
Nov 01 01:05:07.221960 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:07.221924    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:07.228055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:07.228030    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:07.284650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:07.284623    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:07.285756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:07.285740    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:05:09.932948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:09.932915    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 01:05:10.100108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:10.100077    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 01:05:14.934201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:14.934150    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" podUID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" containerName="payload-processing" containerID="cri-o://8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e" gracePeriod=25
Nov 01 01:05:14.989601 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:14.989566    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:05:15.100843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.100800    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" podUID="01fe46e4-f198-4140-a17b-e13cb853ad2b" containerName="payload-pre-processing" containerID="cri-o://7c020b4dd07cff7f62b8c52b50c22dddfb8144e3665df2839253459fa8de7cf9" gracePeriod=25
Nov 01 01:05:15.229627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.229606    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 01:05:15.322279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.322248    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\" (UniqueName: \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\") pod \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") "
Nov 01 01:05:15.322473 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.322374    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\") pod \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\" (UID: \"ce9da717-8e34-4631-94c8-a7c763ca8dc4\") "
Nov 01 01:05:15.322780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.322754    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume" pod "ce9da717-8e34-4631-94c8-a7c763ca8dc4" (UID: "ce9da717-8e34-4631-94c8-a7c763ca8dc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:05:15.325496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.325472    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5" pod "ce9da717-8e34-4631-94c8-a7c763ca8dc4" (UID: "ce9da717-8e34-4631-94c8-a7c763ca8dc4"). InnerVolumeSpecName "kube-api-access-rlrs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:05:15.343202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.343179    4493 generic.go:356] "Generic (PLEG): container finished" podID="01fe46e4-f198-4140-a17b-e13cb853ad2b" containerID="7c020b4dd07cff7f62b8c52b50c22dddfb8144e3665df2839253459fa8de7cf9" exitCode=0
Nov 01 01:05:15.343319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.343248    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" event={"ID":"01fe46e4-f198-4140-a17b-e13cb853ad2b","Type":"ContainerDied","Data":"7c020b4dd07cff7f62b8c52b50c22dddfb8144e3665df2839253459fa8de7cf9"}
Nov 01 01:05:15.344656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.344633    4493 generic.go:356] "Generic (PLEG): container finished" podID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" containerID="8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e" exitCode=0
Nov 01 01:05:15.344744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.344683    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"
Nov 01 01:05:15.344744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.344715    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" event={"ID":"ce9da717-8e34-4631-94c8-a7c763ca8dc4","Type":"ContainerDied","Data":"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"}
Nov 01 01:05:15.344824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.344763    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm" event={"ID":"ce9da717-8e34-4631-94c8-a7c763ca8dc4","Type":"ContainerDied","Data":"20ed81fc35e9161b759230d4a9cbde2972c4523d8620aab32194f33322181f3b"}
Nov 01 01:05:15.344824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.344785    4493 scope.go:122] "RemoveContainer" containerID="8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"
Nov 01 01:05:15.357028 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.357009    4493 scope.go:122] "RemoveContainer" containerID="8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"
Nov 01 01:05:15.357412 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:15.357318    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e\": container with ID starting with 8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e not found: ID does not exist" containerID="8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"
Nov 01 01:05:15.357412 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.357353    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e"} err="failed to get container status \"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e\": rpc error: code = NotFound desc = could not find container \"8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e\": container with ID starting with 8a5ca8eb992618a8497f3f76c411c455d7610df1c9bca60457d83a682935855e not found: ID does not exist"
Nov 01 01:05:15.366762 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.366741    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 01:05:15.423909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.423882    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\") pod \"01fe46e4-f198-4140-a17b-e13cb853ad2b\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") "
Nov 01 01:05:15.424103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.424083    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\" (UniqueName: \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\") pod \"01fe46e4-f198-4140-a17b-e13cb853ad2b\" (UID: \"01fe46e4-f198-4140-a17b-e13cb853ad2b\") "
Nov 01 01:05:15.424339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.424317    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume" pod "01fe46e4-f198-4140-a17b-e13cb853ad2b" (UID: "01fe46e4-f198-4140-a17b-e13cb853ad2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:05:15.424555 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.424537    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9da717-8e34-4631-94c8-a7c763ca8dc4-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:15.424555 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.424556    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fe46e4-f198-4140-a17b-e13cb853ad2b-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:15.424707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.424566    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlrs5\" (UniqueName: \"kubernetes.io/projected/ce9da717-8e34-4631-94c8-a7c763ca8dc4-kube-api-access-rlrs5\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:15.426165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.426146    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf" pod "01fe46e4-f198-4140-a17b-e13cb853ad2b" (UID: "01fe46e4-f198-4140-a17b-e13cb853ad2b"). InnerVolumeSpecName "kube-api-access-2c9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:05:15.525665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.525631    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9xf\" (UniqueName: \"kubernetes.io/projected/01fe46e4-f198-4140-a17b-e13cb853ad2b-kube-api-access-2c9xf\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:15.638023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.637981    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 01:05:15.664565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:15.664522    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-99eb0f-85df449df6-2bbqm"]
Nov 01 01:05:16.371193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.370223    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr" event={"ID":"01fe46e4-f198-4140-a17b-e13cb853ad2b","Type":"ContainerDied","Data":"877e3fa9860ae1f38b9d2b70d2b6e959a85c14d7c3ae45b0b6d64c0ef30c307b"}
Nov 01 01:05:16.371193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.370298    4493 scope.go:122] "RemoveContainer" containerID="7c020b4dd07cff7f62b8c52b50c22dddfb8144e3665df2839253459fa8de7cf9"
Nov 01 01:05:16.371193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.370424    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"
Nov 01 01:05:16.518576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.518538    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" path="/var/lib/kubelet/pods/ce9da717-8e34-4631-94c8-a7c763ca8dc4/volumes"
Nov 01 01:05:16.532732 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.532703    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533295    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" containerName="payload-processing"
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533315    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" containerName="payload-processing"
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533360    4493 cpu_manager.go:414] "containerMap: removing container" podUID="01fe46e4-f198-4140-a17b-e13cb853ad2b" containerName="payload-pre-processing"
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533365    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fe46e4-f198-4140-a17b-e13cb853ad2b" containerName="payload-pre-processing"
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533456    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fe46e4-f198-4140-a17b-e13cb853ad2b" containerName="payload-pre-processing"
Nov 01 01:05:16.533931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.533469    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9da717-8e34-4631-94c8-a7c763ca8dc4" containerName="payload-processing"
Nov 01 01:05:16.538188 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.538171    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.576374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.576335    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 01:05:16.579124 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.579095    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-99eb0f-77d98b7459-rlcfr"]
Nov 01 01:05:16.590614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.590587    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b112c158-serving-cert\""
Nov 01 01:05:16.598491 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.598462    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:05:16.616429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.616399    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:05:16.620821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.620802    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.641519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641491    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.641652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641636    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gnd\" (UniqueName: \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.641727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641686    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.641786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641733    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.641853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641802    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bt7\" (UniqueName: \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.641853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.641834    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.673950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.673922    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-718e19-dockercfg-8kl6g\""
Nov 01 01:05:16.674953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.674926    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-718e19\""
Nov 01 01:05:16.677483 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.677459    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:05:16.743550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743520    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gnd\" (UniqueName: \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.743728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743562    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.743728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743596    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.743728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743614    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bt7\" (UniqueName: \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.743728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743632    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.743728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.743699    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.743991 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:16.743730    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-serving-cert: secret "maas-api-e2e-mt-b112c158-serving-cert" not found
Nov 01 01:05:16.743991 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:16.743800    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls podName:f55abc1d-06b4-47d5-b556-996018072110 nodeName:}" failed. No retries permitted until 2026-11-01 01:05:17.243780085 +0000 UTC m=+2506.921367022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls") pod "maas-api-e2e-mt-b112c158-799d88785c-rxrx8" (UID: "f55abc1d-06b4-47d5-b556-996018072110") : secret "maas-api-e2e-mt-b112c158-serving-cert" not found
Nov 01 01:05:16.744313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.744295    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.744432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.744416    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.746274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.746251    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.770267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.770238    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:05:16.774178 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.774159    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:16.805244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.805151    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gnd\" (UniqueName: \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\") pod \"payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.806267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.806247    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bt7\" (UniqueName: \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:16.831768 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.831730    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:05:16.844418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.844392    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:16.844631 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.844606    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qqz\" (UniqueName: \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:16.939205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.937735    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:16.951629 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.951588    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:16.951921 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.951795    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qqz\" (UniqueName: \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:16.953104 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:16.953075    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:17.027858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.025393    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qqz\" (UniqueName: \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\") pod \"payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:17.089155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.089110    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:17.255630 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.255594    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:17.258123 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.258098    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-rxrx8\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:17.372836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.372814    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:05:17.374488 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:05:17.374461    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c51ee02_ca2a_4baf_85f9_eac144474bae.slice/crio-8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9 WatchSource:0}: Error finding container 8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9: Status 404 returned error can't find the container with id 8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9
Nov 01 01:05:17.383284 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.383260    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" event={"ID":"1c51ee02-ca2a-4baf-85f9-eac144474bae","Type":"ContainerStarted","Data":"8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9"}
Nov 01 01:05:17.452332 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.452309    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:17.481421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.481387    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:05:17.483468 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:05:17.483442    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fc0754_fa9d_4e98_b58f_ea562ff6f3c2.slice/crio-721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4 WatchSource:0}: Error finding container 721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4: Status 404 returned error can't find the container with id 721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4
Nov 01 01:05:17.833855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:17.833833    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:05:17.835526 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:05:17.835498    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55abc1d_06b4_47d5_b556_996018072110.slice/crio-81cde2826aa405f021d3863dd07d42be33d9f7097304ed759d364cce3736397d WatchSource:0}: Error finding container 81cde2826aa405f021d3863dd07d42be33d9f7097304ed759d364cce3736397d: Status 404 returned error can't find the container with id 81cde2826aa405f021d3863dd07d42be33d9f7097304ed759d364cce3736397d
Nov 01 01:05:18.389955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.389921    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" event={"ID":"1c51ee02-ca2a-4baf-85f9-eac144474bae","Type":"ContainerStarted","Data":"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"}
Nov 01 01:05:18.390418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.390118    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:18.391361 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.391338    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" event={"ID":"f55abc1d-06b4-47d5-b556-996018072110","Type":"ContainerStarted","Data":"81cde2826aa405f021d3863dd07d42be33d9f7097304ed759d364cce3736397d"}
Nov 01 01:05:18.392891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.392867    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" event={"ID":"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2","Type":"ContainerStarted","Data":"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"}
Nov 01 01:05:18.392970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.392910    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" event={"ID":"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2","Type":"ContainerStarted","Data":"721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4"}
Nov 01 01:05:18.392970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.392936    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:18.503121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.503092    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fe46e4-f198-4140-a17b-e13cb853ad2b" path="/var/lib/kubelet/pods/01fe46e4-f198-4140-a17b-e13cb853ad2b/volumes"
Nov 01 01:05:18.506962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:18.506918    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" podStartSLOduration=2.506905489 podStartE2EDuration="2.506905489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:05:18.505125524 +0000 UTC m=+2508.182712467" watchObservedRunningTime="2026-11-01 01:05:18.506905489 +0000 UTC m=+2508.184492429"
Nov 01 01:05:19.401511 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:19.401473    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" event={"ID":"f55abc1d-06b4-47d5-b556-996018072110","Type":"ContainerStarted","Data":"79cf8112e3058228a72268be5e35a4f85626d01c7aade5585ca94c1ed3635140"}
Nov 01 01:05:19.401930 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:19.401757    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:19.520560 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:19.520490    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" podStartSLOduration=3.520475115 podStartE2EDuration="3.520475115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:05:18.636188567 +0000 UTC m=+2508.313775513" watchObservedRunningTime="2026-11-01 01:05:19.520475115 +0000 UTC m=+2509.198062060"
Nov 01 01:05:19.520823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:19.520802    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" podStartSLOduration=2.656627724 podStartE2EDuration="3.520793997s" totalImagesPullingTime="864.166273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:05:19.51821469 +0000 UTC m=+2509.195801631" watchObservedRunningTime="2026-11-01 01:05:19.520793997 +0000 UTC m=+2509.198380941"
Nov 01 01:05:23.263476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:23.263445    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:05:25.412714 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:25.412676    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:05:27.031997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.031965    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:05:27.035621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.035604    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.088092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.088059    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-603e8c-serving-cert\""
Nov 01 01:05:27.140595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.140565    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:05:27.171058 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.171018    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwgr\" (UniqueName: \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.171257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.171155    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.171310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.171268    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.171357 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.171318    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.275953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.275912    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwgr\" (UniqueName: \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.276193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.276170    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.276287 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.276258    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.276345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.276313    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.276556 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:27.276532    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-serving-cert: secret "maas-api-e2e-api-a-603e8c-serving-cert" not found
Nov 01 01:05:27.277476 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:27.277459    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls podName:00dc6468-96e7-47bd-8e46-2422ff75e562 nodeName:}" failed. No retries permitted until 2026-11-01 01:05:27.777438098 +0000 UTC m=+2517.455025033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls") pod "maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" (UID: "00dc6468-96e7-47bd-8e46-2422ff75e562") : secret "maas-api-e2e-api-a-603e8c-serving-cert" not found
Nov 01 01:05:27.277994 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.277353    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.306202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.303548    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.340900 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.340868    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwgr\" (UniqueName: \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.782250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.782212    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.784751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.784725    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") pod \"maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:27.948958 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:27.948927    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:28.327521 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:28.327497    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:05:28.329329 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:05:28.329301    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dc6468_96e7_47bd_8e46_2422ff75e562.slice/crio-40d3ccaf1ac633c07fb7f5eea59db8b7639278a0a9f52a0ac6eb84cb4b446bdd WatchSource:0}: Error finding container 40d3ccaf1ac633c07fb7f5eea59db8b7639278a0a9f52a0ac6eb84cb4b446bdd: Status 404 returned error can't find the container with id 40d3ccaf1ac633c07fb7f5eea59db8b7639278a0a9f52a0ac6eb84cb4b446bdd
Nov 01 01:05:28.473909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:28.473873    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" event={"ID":"00dc6468-96e7-47bd-8e46-2422ff75e562","Type":"ContainerStarted","Data":"40d3ccaf1ac633c07fb7f5eea59db8b7639278a0a9f52a0ac6eb84cb4b446bdd"}
Nov 01 01:05:29.405348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:29.405268    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:05:29.405348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:29.405332    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:05:29.481647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:29.481606    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" event={"ID":"00dc6468-96e7-47bd-8e46-2422ff75e562","Type":"ContainerStarted","Data":"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"}
Nov 01 01:05:29.481798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:29.481714    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:29.753324 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:29.753275    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" podStartSLOduration=2.960341553 podStartE2EDuration="3.753260893s" totalImagesPullingTime="792.91934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:05:29.752813621 +0000 UTC m=+2519.430400579" watchObservedRunningTime="2026-11-01 01:05:29.753260893 +0000 UTC m=+2519.430847836"
Nov 01 01:05:35.493143 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:35.493102    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:05:40.322742 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:40.322703    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 01:05:40.323187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:40.322906    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy" containerID="cri-o://7263fabc4058ab9ccc4ae7fb65b195e9673c0316aa1922928c10c0d2101123af" gracePeriod=30
Nov 01 01:05:44.990309 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:44.990272    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:05:45.571658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.571628    4493 generic.go:356] "Generic (PLEG): container finished" podID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerID="7263fabc4058ab9ccc4ae7fb65b195e9673c0316aa1922928c10c0d2101123af" exitCode=0
Nov 01 01:05:45.571835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.571712    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" event={"ID":"4beff93a-f8ea-42c0-864f-151b9c8d4523","Type":"ContainerDied","Data":"7263fabc4058ab9ccc4ae7fb65b195e9673c0316aa1922928c10c0d2101123af"}
Nov 01 01:05:45.603886 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.603810    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 01:05:45.692929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.692878    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.692929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.692935    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.692966    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693010    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693045    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693104    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693245    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693278    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693294    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693374    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:05:45.693468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693442    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\") pod \"4beff93a-f8ea-42c0-864f-151b9c8d4523\" (UID: \"4beff93a-f8ea-42c0-864f-151b9c8d4523\") "
Nov 01 01:05:45.693827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693508    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:05:45.693827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.693723    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:05:45.694064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.694031    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.694064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.694057    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.694200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.694072    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.694200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.694086    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.694200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.694100    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4beff93a-f8ea-42c0-864f-151b9c8d4523-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.695571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.695543    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:05:45.695964 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.695942    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:05:45.696126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.696110    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:05:45.696126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.696152    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr" pod "4beff93a-f8ea-42c0-864f-151b9c8d4523" (UID: "4beff93a-f8ea-42c0-864f-151b9c8d4523"). InnerVolumeSpecName "kube-api-access-bn8dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:05:45.796180 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.796118    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.796333 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.796186    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8dr\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-kube-api-access-bn8dr\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.796333 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.796203    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:45.796333 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:45.796216    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4beff93a-f8ea-42c0-864f-151b9c8d4523-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:46.578478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:46.578447    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"
Nov 01 01:05:46.578478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:46.578457    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t" event={"ID":"4beff93a-f8ea-42c0-864f-151b9c8d4523","Type":"ContainerDied","Data":"cbe2fd17cdf170bcae035a8a6cd8652a29d77ac92d1015aa98d423fbd67dc4d1"}
Nov 01 01:05:46.578949 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:46.578521    4493 scope.go:122] "RemoveContainer" containerID="7263fabc4058ab9ccc4ae7fb65b195e9673c0316aa1922928c10c0d2101123af"
Nov 01 01:05:46.794442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:46.794355    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 01:05:46.817673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:46.817642    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-6wd4t"]
Nov 01 01:05:48.503218 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:48.503190    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" path="/var/lib/kubelet/pods/4beff93a-f8ea-42c0-864f-151b9c8d4523/volumes"
Nov 01 01:05:50.987917 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:50.987886    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 01:05:50.988395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:50.988116    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" podUID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" containerName="maas-api" containerID="cri-o://5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6" gracePeriod=30
Nov 01 01:05:51.268713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.268690    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 01:05:51.360252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360217    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\") pod \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") "
Nov 01 01:05:51.360252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360257    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\" (UniqueName: \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\") pod \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") "
Nov 01 01:05:51.360495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360395    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") pod \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") "
Nov 01 01:05:51.360495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360458    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\") pod \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\" (UID: \"b57b67b9-dda2-46a4-a9b3-f38d49c18f13\") "
Nov 01 01:05:51.360655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca" pod "b57b67b9-dda2-46a4-a9b3-f38d49c18f13" (UID: "b57b67b9-dda2-46a4-a9b3-f38d49c18f13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:05:51.360896 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.360879    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:51.362599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.362576    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs" pod "b57b67b9-dda2-46a4-a9b3-f38d49c18f13" (UID: "b57b67b9-dda2-46a4-a9b3-f38d49c18f13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:05:51.362707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.362685    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls" pod "b57b67b9-dda2-46a4-a9b3-f38d49c18f13" (UID: "b57b67b9-dda2-46a4-a9b3-f38d49c18f13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:05:51.362747 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.362707    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq" pod "b57b67b9-dda2-46a4-a9b3-f38d49c18f13" (UID: "b57b67b9-dda2-46a4-a9b3-f38d49c18f13"). InnerVolumeSpecName "kube-api-access-zvwbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:05:51.461802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.461770    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:51.461802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.461798    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:51.461802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.461808    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwbq\" (UniqueName: \"kubernetes.io/projected/b57b67b9-dda2-46a4-a9b3-f38d49c18f13-kube-api-access-zvwbq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:05:51.608640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.608555    4493 generic.go:356] "Generic (PLEG): container finished" podID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" containerID="5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6" exitCode=0
Nov 01 01:05:51.608640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.608612    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"
Nov 01 01:05:51.608881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.608647    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" event={"ID":"b57b67b9-dda2-46a4-a9b3-f38d49c18f13","Type":"ContainerDied","Data":"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"}
Nov 01 01:05:51.608881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.608711    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v" event={"ID":"b57b67b9-dda2-46a4-a9b3-f38d49c18f13","Type":"ContainerDied","Data":"0f1ac1660ff78a90c8b31ae1e96f540442a5fd0ca08e0d03fe272b465b999afd"}
Nov 01 01:05:51.608881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.608742    4493 scope.go:122] "RemoveContainer" containerID="5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"
Nov 01 01:05:51.618785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.618718    4493 scope.go:122] "RemoveContainer" containerID="5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"
Nov 01 01:05:51.619102 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:05:51.619079    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6\": container with ID starting with 5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6 not found: ID does not exist" containerID="5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"
Nov 01 01:05:51.619192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.619111    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6"} err="failed to get container status \"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6\": rpc error: code = NotFound desc = could not find container \"5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6\": container with ID starting with 5f7c21da22dfd459906520a4688884083c7e69c170804cd46fff0f7d905c72b6 not found: ID does not exist"
Nov 01 01:05:51.774968 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.774936    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 01:05:51.779610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:51.779585    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99eb0f-6fc786c898-vgn7v"]
Nov 01 01:05:52.503427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:05:52.503395    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" path="/var/lib/kubelet/pods/b57b67b9-dda2-46a4-a9b3-f38d49c18f13/volumes"
Nov 01 01:06:02.769066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:02.769030    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:06:02.769614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:02.769255    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" podUID="a4b33793-1b57-441b-acd9-a066d16a9b8f" containerName="maas-api" containerID="cri-o://35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63" gracePeriod=30
Nov 01 01:06:03.112654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.112627    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:06:03.259559 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.259524    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\") pod \"a4b33793-1b57-441b-acd9-a066d16a9b8f\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") "
Nov 01 01:06:03.260589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.260565    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\") pod \"a4b33793-1b57-441b-acd9-a066d16a9b8f\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") "
Nov 01 01:06:03.260862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.260844    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\" (UniqueName: \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\") pod \"a4b33793-1b57-441b-acd9-a066d16a9b8f\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") "
Nov 01 01:06:03.261072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.261055    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") pod \"a4b33793-1b57-441b-acd9-a066d16a9b8f\" (UID: \"a4b33793-1b57-441b-acd9-a066d16a9b8f\") "
Nov 01 01:06:03.265566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.265525    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca" pod "a4b33793-1b57-441b-acd9-a066d16a9b8f" (UID: "a4b33793-1b57-441b-acd9-a066d16a9b8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:06:03.271153 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.271068    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls" pod "a4b33793-1b57-441b-acd9-a066d16a9b8f" (UID: "a4b33793-1b57-441b-acd9-a066d16a9b8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:03.271272 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.271225    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs" pod "a4b33793-1b57-441b-acd9-a066d16a9b8f" (UID: "a4b33793-1b57-441b-acd9-a066d16a9b8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:03.271481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.271449    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx" pod "a4b33793-1b57-441b-acd9-a066d16a9b8f" (UID: "a4b33793-1b57-441b-acd9-a066d16a9b8f"). InnerVolumeSpecName "kube-api-access-qj4rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:03.364797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.364629    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:03.364797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.364665    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b33793-1b57-441b-acd9-a066d16a9b8f-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:03.364797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.364680    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4rx\" (UniqueName: \"kubernetes.io/projected/a4b33793-1b57-441b-acd9-a066d16a9b8f-kube-api-access-qj4rx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:03.364797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.364696    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b33793-1b57-441b-acd9-a066d16a9b8f-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:03.680908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.680827    4493 generic.go:356] "Generic (PLEG): container finished" podID="a4b33793-1b57-441b-acd9-a066d16a9b8f" containerID="35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63" exitCode=0
Nov 01 01:06:03.680908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.680868    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" event={"ID":"a4b33793-1b57-441b-acd9-a066d16a9b8f","Type":"ContainerDied","Data":"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"}
Nov 01 01:06:03.680908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.680904    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt" event={"ID":"a4b33793-1b57-441b-acd9-a066d16a9b8f","Type":"ContainerDied","Data":"4daf6a484c71fb16cdc6afaf0bb55758a72b5b4a967aa741a5c55a69fc86f297"}
Nov 01 01:06:03.681119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.680927    4493 scope.go:122] "RemoveContainer" containerID="35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"
Nov 01 01:06:03.681119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.681043    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"
Nov 01 01:06:03.692263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.692241    4493 scope.go:122] "RemoveContainer" containerID="35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"
Nov 01 01:06:03.692620 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:03.692595    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63\": container with ID starting with 35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63 not found: ID does not exist" containerID="35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"
Nov 01 01:06:03.692701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.692628    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63"} err="failed to get container status \"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63\": rpc error: code = NotFound desc = could not find container \"35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63\": container with ID starting with 35e6df5c7a7aa90bc1264ff234f97e6b7fc1b4c51d95b0963259251a80809a63 not found: ID does not exist"
Nov 01 01:06:03.875093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.875059    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:06:03.880653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:03.880627    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72acf4-7d654dbb47-smqwt"]
Nov 01 01:06:04.509842 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:04.509754    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b33793-1b57-441b-acd9-a066d16a9b8f" path="/var/lib/kubelet/pods/a4b33793-1b57-441b-acd9-a066d16a9b8f/volumes"
Nov 01 01:06:14.990636 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:14.990552    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:06:15.924720 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.924684    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:06:15.925337 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925316    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" containerName="maas-api"
Nov 01 01:06:15.925337 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925337    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" containerName="maas-api"
Nov 01 01:06:15.925512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925364    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy"
Nov 01 01:06:15.925512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925371    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy"
Nov 01 01:06:15.925512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925416    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a4b33793-1b57-441b-acd9-a066d16a9b8f" containerName="maas-api"
Nov 01 01:06:15.925512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925423    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b33793-1b57-441b-acd9-a066d16a9b8f" containerName="maas-api"
Nov 01 01:06:15.925708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925532    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57b67b9-dda2-46a4-a9b3-f38d49c18f13" containerName="maas-api"
Nov 01 01:06:15.925708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925546    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4beff93a-f8ea-42c0-864f-151b9c8d4523" containerName="istio-proxy"
Nov 01 01:06:15.925708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.925564    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b33793-1b57-441b-acd9-a066d16a9b8f" containerName="maas-api"
Nov 01 01:06:15.930244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.930224    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:15.988242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:15.988213    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6b9b61-openshift-default-dockercfg-rvmgf\""
Nov 01 01:06:16.040413 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.040360    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:06:16.043343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043315    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8qs\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043353    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043434    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043520    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043663 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043638    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043673    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043774 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043761    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.043854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.043822    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145391    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145444    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145484    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145530    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145572    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145634    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145676    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8qs\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145708    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.145848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.146054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145880    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.146054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145956    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.146054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.145989    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.146191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.146150    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.146775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.146752    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.148563 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.148541    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.148650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.148641    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.207291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.207225    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.207291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.207271    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8qs\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\") pod \"e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") " pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.243805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.243777    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:16.676234 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.676208    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:06:16.677586 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:16.677563    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298f865e_89c0_4fdd_9c04_2f3aed31619c.slice/crio-eaa1d4459731876581ca2cf1a36db1b6d2fd7dfcac4d7e715cd6ac4cba1f9a1f WatchSource:0}: Error finding container eaa1d4459731876581ca2cf1a36db1b6d2fd7dfcac4d7e715cd6ac4cba1f9a1f: Status 404 returned error can't find the container with id eaa1d4459731876581ca2cf1a36db1b6d2fd7dfcac4d7e715cd6ac4cba1f9a1f
Nov 01 01:06:16.680156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.680107    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:16.680261 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.680200    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:16.680261 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.680247    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:16.791159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.791111    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerStarted","Data":"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"}
Nov 01 01:06:16.791352 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.791195    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerStarted","Data":"eaa1d4459731876581ca2cf1a36db1b6d2fd7dfcac4d7e715cd6ac4cba1f9a1f"}
Nov 01 01:06:16.825459 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.825419    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:06:16.829525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.829501    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:16.881901 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.881877    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-718e19-serving-cert\""
Nov 01 01:06:16.889783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.889755    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:06:16.913572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.913512    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podStartSLOduration=1.9134992199999998 podStartE2EDuration="1.91349922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:16.910503227 +0000 UTC m=+2566.588090172" watchObservedRunningTime="2026-11-01 01:06:16.91349922 +0000 UTC m=+2566.591086165"
Nov 01 01:06:16.955401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.955314    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:16.955401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.955351    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:16.955401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.955378    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmcq\" (UniqueName: \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:16.955612 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:16.955490    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.056877 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.056842    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.057358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.056893    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.057358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.056937    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmcq\" (UniqueName: \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.057358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.056987    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.057358 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:17.057159    4493 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-serving-cert: secret "maas-api-e2e-api-a-718e19-serving-cert" not found
Nov 01 01:06:17.057358 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:17.057231    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls podName:ff57b037-ade1-450a-a893-7d5f1c8bbe0c nodeName:}" failed. No retries permitted until 2026-11-01 01:06:17.557211116 +0000 UTC m=+2567.234798059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls") pod "maas-api-e2e-api-a-718e19-74d6749b96-jpm57" (UID: "ff57b037-ade1-450a-a893-7d5f1c8bbe0c") : secret "maas-api-e2e-api-a-718e19-serving-cert" not found
Nov 01 01:06:17.057699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.057674    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.059771 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.059749    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.146160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.143952    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmcq\" (UniqueName: \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.244447 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.244414    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:17.249834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.249809    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:17.565660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.565571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.568196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.568172    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") pod \"maas-api-e2e-api-a-718e19-74d6749b96-jpm57\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.744705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.744674    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:17.800477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.800447    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:17.802013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:17.801991    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:06:18.132368 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:18.132344    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:06:18.134226 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:18.134194    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff57b037_ade1_450a_a893_7d5f1c8bbe0c.slice/crio-5638e794785645bf0c9eab8db3b04a4844188bc0fd973f102767b71a86b01b31 WatchSource:0}: Error finding container 5638e794785645bf0c9eab8db3b04a4844188bc0fd973f102767b71a86b01b31: Status 404 returned error can't find the container with id 5638e794785645bf0c9eab8db3b04a4844188bc0fd973f102767b71a86b01b31
Nov 01 01:06:18.806763 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:18.806730    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" event={"ID":"ff57b037-ade1-450a-a893-7d5f1c8bbe0c","Type":"ContainerStarted","Data":"5638e794785645bf0c9eab8db3b04a4844188bc0fd973f102767b71a86b01b31"}
Nov 01 01:06:19.259173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.259118    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 01:06:19.309029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.308940    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 01:06:19.311289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.311263    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 01:06:19.816750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.816716    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" event={"ID":"ff57b037-ade1-450a-a893-7d5f1c8bbe0c","Type":"ContainerStarted","Data":"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"}
Nov 01 01:06:19.816937 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.816813    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:19.936209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:19.936111    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" podStartSLOduration=3.082583026 podStartE2EDuration="3.936092862s" totalImagesPullingTime="853.509836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:19.934216072 +0000 UTC m=+2569.611803016" watchObservedRunningTime="2026-11-01 01:06:19.936092862 +0000 UTC m=+2569.613679808"
Nov 01 01:06:23.533626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.533595    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:06:23.537684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.537630    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.598047 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.598016    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:06:23.601988 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.601966    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.647941 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.647910    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:06:23.652312 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652268    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652336    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652475    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652524    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652707    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652766    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652844    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4z4z\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.652995 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.652955    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.653061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.653018    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.668920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.668869    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6b9b61\""
Nov 01 01:06:23.742746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.742711    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6b9b61-dockercfg-dnxkg\""
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774591    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774704    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774749    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774785    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9clh\" (UniqueName: \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774816    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4z4z\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774849    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774907    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774959    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.774986    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.775017    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.775067    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.781156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.776093    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.786415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.782275    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.786415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.782526    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.789262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.789232    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.789517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.789497    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.790453 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.790352    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.791394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.791354    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.798206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.798169    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:06:23.804063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.804037    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:06:23.809906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.809877    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:23.821943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.821902    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:06:23.837440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.837403    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4z4z\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.837606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.837444    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.852593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.851893    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:23.875747 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.875716    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:23.875883 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.875784    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9clh\" (UniqueName: \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.875956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.875884    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.876265 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.876236    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t659m\" (UniqueName: \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:23.876772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.876726    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.938148 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.938096    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9clh\" (UniqueName: \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\") pod \"payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:23.977146 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.977101    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:23.977491 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.977471    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t659m\" (UniqueName: \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:23.978048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:23.978024    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:24.044599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.044525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t659m\" (UniqueName: \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\") pod \"payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:24.125574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.125544    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:24.216423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.216389    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:24.290846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.290819    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:06:24.293949 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:24.293907    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aba9d9f_09d2_4607_9b1c_f8bf4c67408b.slice/crio-bbe5930741af1de73c42a485497ce6f6849d7afab909269cfb55788bd825c5ef WatchSource:0}: Error finding container bbe5930741af1de73c42a485497ce6f6849d7afab909269cfb55788bd825c5ef: Status 404 returned error can't find the container with id bbe5930741af1de73c42a485497ce6f6849d7afab909269cfb55788bd825c5ef
Nov 01 01:06:24.296597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.296565    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:24.296686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.296638    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:24.296686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.296666    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:24.503667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.503639    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:06:24.505387 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:24.505355    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bfcc4a_8362_45cc_b943_b450b214d1df.slice/crio-f89d72cf49ab6d76a25b54f6ee69f9c640a3f7339f675119762f5b76005f2416 WatchSource:0}: Error finding container f89d72cf49ab6d76a25b54f6ee69f9c640a3f7339f675119762f5b76005f2416: Status 404 returned error can't find the container with id f89d72cf49ab6d76a25b54f6ee69f9c640a3f7339f675119762f5b76005f2416
Nov 01 01:06:24.599572 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:24.599545    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba16e7ce_afd5_4491_97d7_40bd174e4f28.slice/crio-abb67cc484f5b1ae761ed3a87253a239b32eb4d4898fffef0f808b89c52983ad WatchSource:0}: Error finding container abb67cc484f5b1ae761ed3a87253a239b32eb4d4898fffef0f808b89c52983ad: Status 404 returned error can't find the container with id abb67cc484f5b1ae761ed3a87253a239b32eb4d4898fffef0f808b89c52983ad
Nov 01 01:06:24.599889 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.599779    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:06:24.848843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.848757    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" event={"ID":"ba16e7ce-afd5-4491-97d7-40bd174e4f28","Type":"ContainerStarted","Data":"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"}
Nov 01 01:06:24.848843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.848810    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" event={"ID":"ba16e7ce-afd5-4491-97d7-40bd174e4f28","Type":"ContainerStarted","Data":"abb67cc484f5b1ae761ed3a87253a239b32eb4d4898fffef0f808b89c52983ad"}
Nov 01 01:06:24.849076 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.848966    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:24.850609 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.850583    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" event={"ID":"a9bfcc4a-8362-45cc-b943-b450b214d1df","Type":"ContainerStarted","Data":"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"}
Nov 01 01:06:24.850757 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.850621    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" event={"ID":"a9bfcc4a-8362-45cc-b943-b450b214d1df","Type":"ContainerStarted","Data":"f89d72cf49ab6d76a25b54f6ee69f9c640a3f7339f675119762f5b76005f2416"}
Nov 01 01:06:24.850757 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.850666    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:24.852337 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.852312    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" event={"ID":"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b","Type":"ContainerStarted","Data":"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"}
Nov 01 01:06:24.852463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.852362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" event={"ID":"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b","Type":"ContainerStarted","Data":"bbe5930741af1de73c42a485497ce6f6849d7afab909269cfb55788bd825c5ef"}
Nov 01 01:06:24.852463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.852391    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:24.968454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:24.968394    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" podStartSLOduration=1.968378531 podStartE2EDuration="1.968378531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:24.964674509 +0000 UTC m=+2574.642261454" watchObservedRunningTime="2026-11-01 01:06:24.968378531 +0000 UTC m=+2574.645965475"
Nov 01 01:06:25.241279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:25.239494    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" podStartSLOduration=2.239472005 podStartE2EDuration="2.239472005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:25.109989319 +0000 UTC m=+2574.787576276" watchObservedRunningTime="2026-11-01 01:06:25.239472005 +0000 UTC m=+2574.917058951"
Nov 01 01:06:25.241279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:25.240478    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podStartSLOduration=2.240463806 podStartE2EDuration="2.240463806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:25.236377609 +0000 UTC m=+2574.913964556" watchObservedRunningTime="2026-11-01 01:06:25.240463806 +0000 UTC m=+2574.918050753"
Nov 01 01:06:25.844898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:25.844862    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:06:25.853948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:25.853909    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:25.854109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:25.853961    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:26.786226 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:26.786192    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 01:06:26.853001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:26.852962    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:26.853401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:26.853020    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:27.853412 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:27.853374    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:27.853832 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:27.853443    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:28.853351 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:28.853315    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:28.853548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:28.853380    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:29.166600 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:29.166518    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:06:29.852529 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:29.852497    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:29.852683 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:29.852573    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:30.852770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:30.852736    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 01 01:06:30.853182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:30.852790    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 01 01:06:31.786588 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:31.786524    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing" containerID="cri-o://86462206f67bc685056c3476fcf92c37c28f45b0b71620ebbd1a6f8b15061c2f" gracePeriod=25
Nov 01 01:06:31.857283 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:31.857253    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:31.857752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:31.857731    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:31.858512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:31.858493    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:06:32.242012 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.241981    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 01:06:32.242285 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.242244    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" podUID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" containerName="maas-api" containerID="cri-o://18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2" gracePeriod=30
Nov 01 01:06:32.590205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.590177    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 01:06:32.635434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.633574    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\" (UniqueName: \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\") pod \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") "
Nov 01 01:06:32.635434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.633649    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") pod \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") "
Nov 01 01:06:32.635434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.633722    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\") pod \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") "
Nov 01 01:06:32.635434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.633852    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\") pod \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\" (UID: \"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe\") "
Nov 01 01:06:32.644874 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.644824    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls" pod "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" (UID: "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:32.645202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.645165    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs" pod "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" (UID: "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:32.645496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.645473    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca" pod "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" (UID: "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:06:32.666871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.660379    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd" pod "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" (UID: "b6e8b9f2-3483-4e53-b08a-f4b94c908dfe"). InnerVolumeSpecName "kube-api-access-8jvdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:32.735603 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.735563    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:32.735603 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.735595    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:32.735603 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.735608    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvdd\" (UniqueName: \"kubernetes.io/projected/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-kube-api-access-8jvdd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:32.735841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.735622    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:32.901510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.901430    4493 generic.go:356] "Generic (PLEG): container finished" podID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" containerID="18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2" exitCode=0
Nov 01 01:06:32.901510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.901477    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"
Nov 01 01:06:32.901954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.901516    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" event={"ID":"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe","Type":"ContainerDied","Data":"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"}
Nov 01 01:06:32.901954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.901573    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q" event={"ID":"b6e8b9f2-3483-4e53-b08a-f4b94c908dfe","Type":"ContainerDied","Data":"f20cb95b49065db76efe2fbacf6228fdcb9dc462a027dbae1e971d1fb7636e8f"}
Nov 01 01:06:32.901954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.901601    4493 scope.go:122] "RemoveContainer" containerID="18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"
Nov 01 01:06:32.927674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.927558    4493 scope.go:122] "RemoveContainer" containerID="18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"
Nov 01 01:06:32.928035 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:32.928014    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2\": container with ID starting with 18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2 not found: ID does not exist" containerID="18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"
Nov 01 01:06:32.928177 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:32.928042    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2"} err="failed to get container status \"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2\": rpc error: code = NotFound desc = could not find container \"18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2\": container with ID starting with 18bfe92d4cff59f9b229b70311431668b3b87e61e8d15773018b6f3a3992b2f2 not found: ID does not exist"
Nov 01 01:06:33.074197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:33.074155    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 01:06:33.079843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:33.079820    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97e89-6f869b9df9-hht5q"]
Nov 01 01:06:33.278391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:33.278353    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:06:34.280546 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.280475    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main" containerID="cri-o://1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd" gracePeriod=45
Nov 01 01:06:34.334874 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.332072    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main" containerID="cri-o://595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8" gracePeriod=45
Nov 01 01:06:34.337936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.337763    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main" containerID="cri-o://11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf" gracePeriod=45
Nov 01 01:06:34.505804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.505764    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" path="/var/lib/kubelet/pods/b6e8b9f2-3483-4e53-b08a-f4b94c908dfe/volumes"
Nov 01 01:06:34.669165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.665072    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 01:06:34.670514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.670287    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 01:06:34.674039 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.674022    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 01:06:34.767123 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767092    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.767353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767167    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.767353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767215    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.767353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767245    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\" (UniqueName: \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.767353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767271    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767365    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\" (UniqueName: \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767419    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\" (UniqueName: \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767398    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767450    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767482    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767488    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.767531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767523    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.767904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767600    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.768036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767644    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.768170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768074    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.768170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768113    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\") pod \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\" (UID: \"8cba285c-f5c9-4a51-8e8c-1bead9e00439\") "
Nov 01 01:06:34.768320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768212    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.768320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768246    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.768320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768289    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\") pod \"b6408e85-6f99-44ec-865c-c721fa95f958\" (UID: \"b6408e85-6f99-44ec-865c-c721fa95f958\") "
Nov 01 01:06:34.768488 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768325    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\" (UniqueName: \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\") pod \"923b9d82-3b07-4d40-99ef-2a0928488985\" (UID: \"923b9d82-3b07-4d40-99ef-2a0928488985\") "
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767799    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.767888    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768024    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.768063    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.769068    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.769157    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.769356    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.769454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.769426    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.770161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.769515    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.770361 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.770338    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.770489 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.770464    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "kube-api-access-jp64t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:34.771025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.771004    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs" pod "b6408e85-6f99-44ec-865c-c721fa95f958" (UID: "b6408e85-6f99-44ec-865c-c721fa95f958"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:34.771931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.771851    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:34.771931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.771916    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "kube-api-access-brvkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:34.771931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.771942    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "kube-api-access-qj2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:34.772406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.772383    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm" pod "8cba285c-f5c9-4a51-8e8c-1bead9e00439" (UID: "8cba285c-f5c9-4a51-8e8c-1bead9e00439"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.772490 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.772383    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:06:34.773324 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.773306    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs" pod "923b9d82-3b07-4d40-99ef-2a0928488985" (UID: "923b9d82-3b07-4d40-99ef-2a0928488985"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870628    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp64t\" (UniqueName: \"kubernetes.io/projected/b6408e85-6f99-44ec-865c-c721fa95f958-kube-api-access-jp64t\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870657    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2bj\" (UniqueName: \"kubernetes.io/projected/8cba285c-f5c9-4a51-8e8c-1bead9e00439-kube-api-access-qj2bj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870668    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cba285c-f5c9-4a51-8e8c-1bead9e00439-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870677    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870685    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870695    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870703    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6408e85-6f99-44ec-865c-c721fa95f958-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.870705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870711    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870720    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cba285c-f5c9-4a51-8e8c-1bead9e00439-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870728    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870736    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870744    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870751    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923b9d82-3b07-4d40-99ef-2a0928488985-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870758    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6408e85-6f99-44ec-865c-c721fa95f958-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870767    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvkz\" (UniqueName: \"kubernetes.io/projected/923b9d82-3b07-4d40-99ef-2a0928488985-kube-api-access-brvkz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.871055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.870777    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923b9d82-3b07-4d40-99ef-2a0928488985-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:34.923220 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.923182    4493 generic.go:356] "Generic (PLEG): container finished" podID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerID="11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf" exitCode=0
Nov 01 01:06:34.923445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.923286    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"
Nov 01 01:06:34.923445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.923324    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" event={"ID":"8cba285c-f5c9-4a51-8e8c-1bead9e00439","Type":"ContainerDied","Data":"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"}
Nov 01 01:06:34.923445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.923386    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz" event={"ID":"8cba285c-f5c9-4a51-8e8c-1bead9e00439","Type":"ContainerDied","Data":"532a29c70c1e9b5424478fb83aaf42b8731fd39567c5b97f71f8706dc5cef7fa"}
Nov 01 01:06:34.923445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.923408    4493 scope.go:122] "RemoveContainer" containerID="11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"
Nov 01 01:06:34.924922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.924838    4493 generic.go:356] "Generic (PLEG): container finished" podID="b6408e85-6f99-44ec-865c-c721fa95f958" containerID="1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd" exitCode=0
Nov 01 01:06:34.925049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.925001    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" event={"ID":"b6408e85-6f99-44ec-865c-c721fa95f958","Type":"ContainerDied","Data":"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"}
Nov 01 01:06:34.925049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.925034    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf" event={"ID":"b6408e85-6f99-44ec-865c-c721fa95f958","Type":"ContainerDied","Data":"95eebcf0b285d1ee37d3cda800e89ce4a4afbbc07f07166cad861cbc31e81e50"}
Nov 01 01:06:34.925294 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.925093    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"
Nov 01 01:06:34.933524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.926871    4493 generic.go:356] "Generic (PLEG): container finished" podID="923b9d82-3b07-4d40-99ef-2a0928488985" containerID="595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8" exitCode=0
Nov 01 01:06:34.933524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.926921    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" event={"ID":"923b9d82-3b07-4d40-99ef-2a0928488985","Type":"ContainerDied","Data":"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"}
Nov 01 01:06:34.933524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.926949    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2" event={"ID":"923b9d82-3b07-4d40-99ef-2a0928488985","Type":"ContainerDied","Data":"0021bf6aa53d351a8d6f8f9c217096259acc216e00c003bf7f6e5d955621c6fa"}
Nov 01 01:06:34.933524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.927008    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"
Nov 01 01:06:34.941637 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.941613    4493 scope.go:122] "RemoveContainer" containerID="11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"
Nov 01 01:06:34.942102 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:34.942073    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf\": container with ID starting with 11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf not found: ID does not exist" containerID="11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"
Nov 01 01:06:34.942230 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.942112    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf"} err="failed to get container status \"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf\": rpc error: code = NotFound desc = could not find container \"11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf\": container with ID starting with 11297dc22fd58897d572e43df4ce748347aabf7ce206f0eb7301edc1a88375bf not found: ID does not exist"
Nov 01 01:06:34.942230 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.942203    4493 scope.go:122] "RemoveContainer" containerID="1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"
Nov 01 01:06:34.956588 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.956566    4493 scope.go:122] "RemoveContainer" containerID="1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"
Nov 01 01:06:34.956913 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:34.956893    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd\": container with ID starting with 1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd not found: ID does not exist" containerID="1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"
Nov 01 01:06:34.956975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.956922    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd"} err="failed to get container status \"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd\": rpc error: code = NotFound desc = could not find container \"1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd\": container with ID starting with 1c639132cc8e31dc3ff2d6c619a9fe0298802426e99118dc6800b0c981fe69bd not found: ID does not exist"
Nov 01 01:06:34.956975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.956945    4493 scope.go:122] "RemoveContainer" containerID="595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"
Nov 01 01:06:34.965761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.965737    4493 scope.go:122] "RemoveContainer" containerID="595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"
Nov 01 01:06:34.966080 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:34.966059    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8\": container with ID starting with 595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8 not found: ID does not exist" containerID="595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"
Nov 01 01:06:34.966190 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:34.966089    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8"} err="failed to get container status \"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8\": rpc error: code = NotFound desc = could not find container \"595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8\": container with ID starting with 595c3eead46ec09aa46399dd399bcadd4d23788666266d8a5f26d1421ad87aa8 not found: ID does not exist"
Nov 01 01:06:35.176194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.176093    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 01:06:35.181054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.181024    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-2-w2-4143b7-kserve-778c487cd9-qsknf"]
Nov 01 01:06:35.289989 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.289947    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 01:06:35.297751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.297721    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/simulator-w2-4143b7-kserve-5fcf566f7d-hbzpz"]
Nov 01 01:06:35.410196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.410163    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 01:06:35.414064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.414040    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/distinct-w2-4143b7-kserve-5bd6dfb894-xk9k2"]
Nov 01 01:06:35.860416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.860385    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:06:35.860601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:35.860453    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:06:36.503663 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:36.503635    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" path="/var/lib/kubelet/pods/8cba285c-f5c9-4a51-8e8c-1bead9e00439/volumes"
Nov 01 01:06:36.504054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:36.504039    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" path="/var/lib/kubelet/pods/923b9d82-3b07-4d40-99ef-2a0928488985/volumes"
Nov 01 01:06:36.504446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:36.504433    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" path="/var/lib/kubelet/pods/b6408e85-6f99-44ec-865c-c721fa95f958/volumes"
Nov 01 01:06:38.300648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.300614    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7113-508fd8"]
Nov 01 01:06:38.301200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301185    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main"
Nov 01 01:06:38.301200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301201    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301223    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301227    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301244    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" containerName="maas-api"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301252    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" containerName="maas-api"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301271    4493 cpu_manager.go:414] "containerMap: removing container" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main"
Nov 01 01:06:38.301288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301276    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main"
Nov 01 01:06:38.301467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301368    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="923b9d82-3b07-4d40-99ef-2a0928488985" containerName="main"
Nov 01 01:06:38.301467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301378    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cba285c-f5c9-4a51-8e8c-1bead9e00439" containerName="main"
Nov 01 01:06:38.301467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301390    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6408e85-6f99-44ec-865c-c721fa95f958" containerName="main"
Nov 01 01:06:38.301467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.301397    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e8b9f2-3483-4e53-b08a-f4b94c908dfe" containerName="maas-api"
Nov 01 01:06:38.306095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.306077    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:38.358696 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.358667    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-508fd8"]
Nov 01 01:06:38.404967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.404936    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2k2hw\""
Nov 01 01:06:38.413905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.413881    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lhk\" (UniqueName: \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\") pod \"test-curl-7113-508fd8\" (UID: \"f9bcc100-2b35-4730-b99c-470d930703a3\") " pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:38.515432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.515394    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lhk\" (UniqueName: \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\") pod \"test-curl-7113-508fd8\" (UID: \"f9bcc100-2b35-4730-b99c-470d930703a3\") " pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:38.576785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.576705    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lhk\" (UniqueName: \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\") pod \"test-curl-7113-508fd8\" (UID: \"f9bcc100-2b35-4730-b99c-470d930703a3\") " pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:38.618387 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.618355    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:38.666465 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.666435    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:06:38.670442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.670421    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.724756    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.724796    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.724860    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.724889    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.724912    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.725005    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.725066    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6hp\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.725103    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.726160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.725225    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.826679 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.826630    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6hp\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.826909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.826696    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.826909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.826811    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.826943    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.826994    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827032    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827063    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827103    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827163    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827600 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827565    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827633    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827696    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827725    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.827998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.827968    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.829607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.829585    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.830044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.830026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.889658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.889624    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:38.891682 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.891380    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:06:38.892236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:38.892211    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6hp\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:39.002625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.002601    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-508fd8"]
Nov 01 01:06:39.004315 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.004293    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:39.004527 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:39.004506    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9bcc100_2b35_4730_b99c_470d930703a3.slice/crio-299b2d16317ea3102958dd08340abf6b900a179d0e414ae5ab63b6a5ccdab34e WatchSource:0}: Error finding container 299b2d16317ea3102958dd08340abf6b900a179d0e414ae5ab63b6a5ccdab34e: Status 404 returned error can't find the container with id 299b2d16317ea3102958dd08340abf6b900a179d0e414ae5ab63b6a5ccdab34e
Nov 01 01:06:39.451214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.451178    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:06:39.472080 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:39.470952    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fc6cd1_3a42_4283_9e71_9540c51e37b6.slice/crio-3b8630607e841201d99cb4c94c2eaa9d8c5005d456c2c5eadf51dbb13847e533 WatchSource:0}: Error finding container 3b8630607e841201d99cb4c94c2eaa9d8c5005d456c2c5eadf51dbb13847e533: Status 404 returned error can't find the container with id 3b8630607e841201d99cb4c94c2eaa9d8c5005d456c2c5eadf51dbb13847e533
Nov 01 01:06:39.475794 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.475755    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:39.475919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.475829    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:39.475919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.475859    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:39.960800 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.960761    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-508fd8" event={"ID":"f9bcc100-2b35-4730-b99c-470d930703a3","Type":"ContainerStarted","Data":"299b2d16317ea3102958dd08340abf6b900a179d0e414ae5ab63b6a5ccdab34e"}
Nov 01 01:06:39.962654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.962626    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" event={"ID":"11fc6cd1-3a42-4283-9e71-9540c51e37b6","Type":"ContainerStarted","Data":"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"}
Nov 01 01:06:39.962782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:39.962672    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" event={"ID":"11fc6cd1-3a42-4283-9e71-9540c51e37b6","Type":"ContainerStarted","Data":"3b8630607e841201d99cb4c94c2eaa9d8c5005d456c2c5eadf51dbb13847e533"}
Nov 01 01:06:40.004580 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:40.004547    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:40.006288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:40.006258    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:40.006404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:40.006324    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:40.084458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:40.084402    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podStartSLOduration=2.084387659 podStartE2EDuration="2.084387659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:40.083742597 +0000 UTC m=+2589.761329546" watchObservedRunningTime="2026-11-01 01:06:40.084387659 +0000 UTC m=+2589.761974615"
Nov 01 01:06:40.969359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:40.969323    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-508fd8" event={"ID":"f9bcc100-2b35-4730-b99c-470d930703a3","Type":"ContainerStarted","Data":"2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376"}
Nov 01 01:06:41.004987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:41.004950    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:41.005183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:41.005007    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:41.095252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:41.095189    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7113-508fd8" podStartSLOduration=1.837630607 podStartE2EDuration="3.095170137s" totalImagesPullingTime="1.25753953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:41.094319799 +0000 UTC m=+2590.771906746" watchObservedRunningTime="2026-11-01 01:06:41.095170137 +0000 UTC m=+2590.772757080"
Nov 01 01:06:42.005183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:42.005111    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:42.005570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:42.005213    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:42.746567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:42.746525    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7113-508fd8"]
Nov 01 01:06:42.748611 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:42.748582    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7113-508fd8"]
Nov 01 01:06:42.981470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:42.981431    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7113-508fd8" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" containerName="test-curl-7113-508fd8" containerID="cri-o://2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376" gracePeriod=2
Nov 01 01:06:43.005620 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:43.005521    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:43.005971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:43.005635    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:43.038425 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:43.038378    4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-508fd8\" is forbidden: User \"system:node:ip-10-0-141-75.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-75.us-west-1.compute.internal' and this object" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:43.277880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:43.277794    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:06:44.004875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:44.004840    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:44.005055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:44.004894    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:44.993862 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:44.993832    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:06:45.005565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.005533    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 01:06:45.005641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.005606    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 01:06:45.255773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.255710    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:45.305595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.305559    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\" (UniqueName: \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\") pod \"f9bcc100-2b35-4730-b99c-470d930703a3\" (UID: \"f9bcc100-2b35-4730-b99c-470d930703a3\") "
Nov 01 01:06:45.307994 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.307965    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk" pod "f9bcc100-2b35-4730-b99c-470d930703a3" (UID: "f9bcc100-2b35-4730-b99c-470d930703a3"). InnerVolumeSpecName "kube-api-access-z5lhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:45.406954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:45.406918    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lhk\" (UniqueName: \"kubernetes.io/projected/f9bcc100-2b35-4730-b99c-470d930703a3-kube-api-access-z5lhk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:46.009201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.009161    4493 generic.go:356] "Generic (PLEG): container finished" podID="f9bcc100-2b35-4730-b99c-470d930703a3" containerID="2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376" exitCode=137
Nov 01 01:06:46.009729 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.009216    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:46.009729 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.009246    4493 scope.go:122] "RemoveContainer" containerID="2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376"
Nov 01 01:06:46.020917 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.020841    4493 scope.go:122] "RemoveContainer" containerID="2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376"
Nov 01 01:06:46.021280 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:46.021252    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376\": container with ID starting with 2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376 not found: ID does not exist" containerID="2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376"
Nov 01 01:06:46.021386 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.021289    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376"} err="failed to get container status \"2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376\": rpc error: code = NotFound desc = could not find container \"2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376\": container with ID starting with 2c5ca20d63126004c54cb00ec3c80628c5b3aeaf87e62fcf7ac26f1f3f58c376 not found: ID does not exist"
Nov 01 01:06:46.073191 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:46.073120    4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-508fd8\" is forbidden: User \"system:node:ip-10-0-141-75.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-75.us-west-1.compute.internal' and this object" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:46.145264 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.145225    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:46.145856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.145825    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:46.146783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.146767    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:06:46.319127 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:46.319039    4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-508fd8\" is forbidden: User \"system:node:ip-10-0-141-75.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-75.us-west-1.compute.internal' and this object" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:46.491663 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:06:46.491620    4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-508fd8\" is forbidden: User \"system:node:ip-10-0-141-75.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-75.us-west-1.compute.internal' and this object" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" pod="opendatahub/test-curl-7113-508fd8"
Nov 01 01:06:46.502876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:46.502847    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" path="/var/lib/kubelet/pods/f9bcc100-2b35-4730-b99c-470d930703a3/volumes"
Nov 01 01:06:51.670158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.670103    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:06:51.670735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.670718    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" containerName="test-curl-7113-508fd8"
Nov 01 01:06:51.670785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.670737    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" containerName="test-curl-7113-508fd8"
Nov 01 01:06:51.670833 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.670824    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bcc100-2b35-4730-b99c-470d930703a3" containerName="test-curl-7113-508fd8"
Nov 01 01:06:51.675466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.675448    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.751155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.739470    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-030da8-openshift-default-dockercfg-jvk72\""
Nov 01 01:06:51.787485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.787344    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.787797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.787771    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.787987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.787973    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgmj\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.788274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.788253    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.788463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.788442    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.788664 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.788648    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.788929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.788910    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.789099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.789082    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.789237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.789219    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.812033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.811983    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:06:51.896929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.896890    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.896952    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897044    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgmj\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897268 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897173    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897268 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897223    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897294    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.897523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.897395    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.898990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.898937    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.898990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.898976    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.901440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.899488    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.902038 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.901891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.906768 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.906742    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.907235 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.907213    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.908591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.908568    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.958938 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.958862    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:51.959099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:51.959028    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgmj\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\") pod \"e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") " pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:52.030869 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.030836    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:52.050458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.050431    4493 generic.go:356] "Generic (PLEG): container finished" podID="7dbc4516-586a-4530-811e-689ce371b521" containerID="86462206f67bc685056c3476fcf92c37c28f45b0b71620ebbd1a6f8b15061c2f" exitCode=0
Nov 01 01:06:52.050595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.050507    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" event={"ID":"7dbc4516-586a-4530-811e-689ce371b521","Type":"ContainerDied","Data":"86462206f67bc685056c3476fcf92c37c28f45b0b71620ebbd1a6f8b15061c2f"}
Nov 01 01:06:52.063486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.063462    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 01:06:52.205215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.205186    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\") pod \"7dbc4516-586a-4530-811e-689ce371b521\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") "
Nov 01 01:06:52.205406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.205241    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\" (UniqueName: \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\") pod \"7dbc4516-586a-4530-811e-689ce371b521\" (UID: \"7dbc4516-586a-4530-811e-689ce371b521\") "
Nov 01 01:06:52.205674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.205648    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume" pod "7dbc4516-586a-4530-811e-689ce371b521" (UID: "7dbc4516-586a-4530-811e-689ce371b521"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:06:52.205781 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.205764    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbc4516-586a-4530-811e-689ce371b521-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:52.207547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.207525    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c" pod "7dbc4516-586a-4530-811e-689ce371b521" (UID: "7dbc4516-586a-4530-811e-689ce371b521"). InnerVolumeSpecName "kube-api-access-llq8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:06:52.306710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.306670    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llq8c\" (UniqueName: \"kubernetes.io/projected/7dbc4516-586a-4530-811e-689ce371b521-kube-api-access-llq8c\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:06:52.499197 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:52.499168    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ede110_50e6_45fe_a56a_ad1bbeefe5d0.slice/crio-0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad WatchSource:0}: Error finding container 0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad: Status 404 returned error can't find the container with id 0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad
Nov 01 01:06:52.501821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.501785    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:52.501911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.501855    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:52.501911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.501885    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:52.502846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:52.502829    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:06:53.058381 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.058336    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerStarted","Data":"d40ab96fe59ebe104d1023e7b5735c7471e13e0b9b31b372c3bbd8c1fc650026"}
Nov 01 01:06:53.058872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.058412    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerStarted","Data":"0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad"}
Nov 01 01:06:53.059791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.059773    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"
Nov 01 01:06:53.059893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.059799    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm" event={"ID":"7dbc4516-586a-4530-811e-689ce371b521","Type":"ContainerDied","Data":"1606eecee975f5d4ac9c9b418e7c33e13ec68fe77b6af942326d5e5d370db57c"}
Nov 01 01:06:53.059893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.059851    4493 scope.go:122] "RemoveContainer" containerID="86462206f67bc685056c3476fcf92c37c28f45b0b71620ebbd1a6f8b15061c2f"
Nov 01 01:06:53.181567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.181501    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podStartSLOduration=2.181478911 podStartE2EDuration="2.181478911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:53.178526645 +0000 UTC m=+2602.856113594" watchObservedRunningTime="2026-11-01 01:06:53.181478911 +0000 UTC m=+2602.859065857"
Nov 01 01:06:53.296588 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.292561    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 01:06:53.321223 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.314554    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4143b7-98fcfc859-2vxjm"]
Nov 01 01:06:53.779285 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.779243    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:06:53.780526 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.780493    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing"
Nov 01 01:06:53.780526 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.780525    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing"
Nov 01 01:06:53.780862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.780845    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbc4516-586a-4530-811e-689ce371b521" containerName="payload-pre-processing"
Nov 01 01:06:53.787054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.787029    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.871891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.871509    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:06:53.928456 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928409    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928475    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928567    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928614    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928722    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928771    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928806    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxv9\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928872    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:53.928939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:53.928916    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030219    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030270    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030298    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030355    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030382    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030449    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxv9\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030486    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030535    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.030882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.030840    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.031023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.031001    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.031068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.031039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.031258 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.031115    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.031357 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.031342    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:54.033011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.032981    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.033175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.033111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.038050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.037992    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:54.066864 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.066835    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:54.067834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.067814    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:06:54.091963 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.091927    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.092092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.092043    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxv9\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.103106 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.103060    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:54.502874 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.502842    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbc4516-586a-4530-811e-689ce371b521" path="/var/lib/kubelet/pods/7dbc4516-586a-4530-811e-689ce371b521/volumes"
Nov 01 01:06:54.543950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.543922    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:06:54.546187 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:06:54.546158    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b92ba3_60ed_4a0e_8067_7daf614d6e7c.slice/crio-e9065cd7337775f4b2f040d7480339711ffebff8a9d254f023dbea8572508a27 WatchSource:0}: Error finding container e9065cd7337775f4b2f040d7480339711ffebff8a9d254f023dbea8572508a27: Status 404 returned error can't find the container with id e9065cd7337775f4b2f040d7480339711ffebff8a9d254f023dbea8572508a27
Nov 01 01:06:54.548809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.548780    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:54.548880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.548863    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:54.548922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:54.548894    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:06:55.074701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.074660    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" event={"ID":"28b92ba3-60ed-4a0e-8067-7daf614d6e7c","Type":"ContainerStarted","Data":"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"}
Nov 01 01:06:55.075221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.074733    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" event={"ID":"28b92ba3-60ed-4a0e-8067-7daf614d6e7c","Type":"ContainerStarted","Data":"e9065cd7337775f4b2f040d7480339711ffebff8a9d254f023dbea8572508a27"}
Nov 01 01:06:55.104232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.104195    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:06:55.106099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.106075    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:06:55.106191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.106150    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:06:55.204077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:55.204018    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podStartSLOduration=2.20400168 podStartE2EDuration="2.20400168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:55.199828942 +0000 UTC m=+2604.877415888" watchObservedRunningTime="2026-11-01 01:06:55.20400168 +0000 UTC m=+2604.881588624"
Nov 01 01:06:56.103525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:56.103487    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:06:56.104063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:56.103550    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:06:57.103913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:57.103880    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:06:57.104459 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:57.103937    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:06:58.103643 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:58.103607    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:06:58.103838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:58.103660    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:06:59.104024 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:59.103987    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:06:59.104446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:06:59.104050    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:07:00.104403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:00.104365    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 01:07:00.104886 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:00.104440    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 01:07:01.244855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.244824    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:07:01.249186 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.249162    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.306917 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.306880    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:07:01.383158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.375467    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-030da8\""
Nov 01 01:07:01.383158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.375551    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-030da8-dockercfg-p2z7m\""
Nov 01 01:07:01.398155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.392151    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.398155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.392210    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f57v\" (UniqueName: \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.415001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.414962    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:07:01.442712 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.442680    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.464642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.464386    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:07:01.495275 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.495188    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.495275 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.495238    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f57v\" (UniqueName: \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.496384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.496359    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.556783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.556751    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f57v\" (UniqueName: \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\") pod \"payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.564509 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.564462    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:01.597092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.597057    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wgf\" (UniqueName: \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.597332 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.597309    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.698985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.698946    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.699197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.699149    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wgf\" (UniqueName: \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.699911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.699875    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.761079 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.761004    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wgf\" (UniqueName: \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\") pod \"payload-processing-e2e-api-b-030da8-6db6f68668-wszfg\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") " pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:01.817101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.817069    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:07:01.817612 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.817544    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:07:01.818575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.818556    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:07:01.950700 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:01.950663    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:07:01.951123 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:07:01.951101    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd55d2f0_a035_46dd_989f_cbc1063bc253.slice/crio-9e2a4353c7807898f9cdc863c4475089837def292168b7e0b7d3705665cf375d WatchSource:0}: Error finding container 9e2a4353c7807898f9cdc863c4475089837def292168b7e0b7d3705665cf375d: Status 404 returned error can't find the container with id 9e2a4353c7807898f9cdc863c4475089837def292168b7e0b7d3705665cf375d
Nov 01 01:07:02.059670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.059631    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:02.207378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.207343    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" event={"ID":"dd55d2f0-a035-46dd-989f-cbc1063bc253","Type":"ContainerStarted","Data":"09aec913edc20c7f0d474fbf202285b13926614f5f476ad6b9379b4ab8cccf4a"}
Nov 01 01:07:02.207572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.207399    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" event={"ID":"dd55d2f0-a035-46dd-989f-cbc1063bc253","Type":"ContainerStarted","Data":"9e2a4353c7807898f9cdc863c4475089837def292168b7e0b7d3705665cf375d"}
Nov 01 01:07:02.207572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.207568    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:02.324464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.324346    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" podStartSLOduration=1.32432847 podStartE2EDuration="1.32432847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:02.322412439 +0000 UTC m=+2611.999999385" watchObservedRunningTime="2026-11-01 01:07:02.32432847 +0000 UTC m=+2612.001915415"
Nov 01 01:07:02.472461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:02.471238    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:07:03.214520 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:03.214487    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" event={"ID":"bc099aae-5b39-494e-8a68-ec4b8ff3743e","Type":"ContainerStarted","Data":"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"}
Nov 01 01:07:03.214698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:03.214539    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" event={"ID":"bc099aae-5b39-494e-8a68-ec4b8ff3743e","Type":"ContainerStarted","Data":"753aff6c58cc6ac585f21d9637c5ed9f669a774e62fda7726eaae846e818bebd"}
Nov 01 01:07:03.214785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:03.214773    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:03.331394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:03.331348    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" podStartSLOduration=2.331333969 podStartE2EDuration="2.331333969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:03.329613271 +0000 UTC m=+2613.007200216" watchObservedRunningTime="2026-11-01 01:07:03.331333969 +0000 UTC m=+2613.008920891"
Nov 01 01:07:05.468194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.468150    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 01:07:05.468686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.468436    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" podUID="f50e39bf-f364-4fc1-b958-9a5a251d0804" containerName="maas-api" containerID="cri-o://725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0" gracePeriod=30
Nov 01 01:07:05.779631 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.779605    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 01:07:05.858779 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.858745    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\") pod \"f50e39bf-f364-4fc1-b958-9a5a251d0804\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") "
Nov 01 01:07:05.858993 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.858881    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\" (UniqueName: \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\") pod \"f50e39bf-f364-4fc1-b958-9a5a251d0804\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") "
Nov 01 01:07:05.858993 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.858990    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\") pod \"f50e39bf-f364-4fc1-b958-9a5a251d0804\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") "
Nov 01 01:07:05.859102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.859053    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\") pod \"f50e39bf-f364-4fc1-b958-9a5a251d0804\" (UID: \"f50e39bf-f364-4fc1-b958-9a5a251d0804\") "
Nov 01 01:07:05.859755 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.859727    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca" pod "f50e39bf-f364-4fc1-b958-9a5a251d0804" (UID: "f50e39bf-f364-4fc1-b958-9a5a251d0804"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:05.861202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.861176    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs" pod "f50e39bf-f364-4fc1-b958-9a5a251d0804" (UID: "f50e39bf-f364-4fc1-b958-9a5a251d0804"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:05.861347 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.861325    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls" pod "f50e39bf-f364-4fc1-b958-9a5a251d0804" (UID: "f50e39bf-f364-4fc1-b958-9a5a251d0804"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:05.861443 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.861426    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2" pod "f50e39bf-f364-4fc1-b958-9a5a251d0804" (UID: "f50e39bf-f364-4fc1-b958-9a5a251d0804"). InnerVolumeSpecName "kube-api-access-kfsp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:05.976173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.976103    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:05.976173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.976156    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50e39bf-f364-4fc1-b958-9a5a251d0804-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:05.976173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.976171    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f50e39bf-f364-4fc1-b958-9a5a251d0804-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:05.976709 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:05.976185    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfsp2\" (UniqueName: \"kubernetes.io/projected/f50e39bf-f364-4fc1-b958-9a5a251d0804-kube-api-access-kfsp2\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:06.235325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.235293    4493 generic.go:356] "Generic (PLEG): container finished" podID="f50e39bf-f364-4fc1-b958-9a5a251d0804" containerID="725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0" exitCode=0
Nov 01 01:07:06.235541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.235348    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"
Nov 01 01:07:06.235541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.235386    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" event={"ID":"f50e39bf-f364-4fc1-b958-9a5a251d0804","Type":"ContainerDied","Data":"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"}
Nov 01 01:07:06.235541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.235462    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl" event={"ID":"f50e39bf-f364-4fc1-b958-9a5a251d0804","Type":"ContainerDied","Data":"96fb6079a204850dbbe8e160735cfc010930923584a30c8e89f3ee322ab34eb1"}
Nov 01 01:07:06.235541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.235495    4493 scope.go:122] "RemoveContainer" containerID="725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"
Nov 01 01:07:06.245975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.245957    4493 scope.go:122] "RemoveContainer" containerID="725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"
Nov 01 01:07:06.246278 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:06.246256    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0\": container with ID starting with 725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0 not found: ID does not exist" containerID="725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"
Nov 01 01:07:06.246370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.246285    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0"} err="failed to get container status \"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0\": rpc error: code = NotFound desc = could not find container \"725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0\": container with ID starting with 725fdc246e78dbc5f06eddc5b7483c81b9a641f2bb307ea76c99433558d757c0 not found: ID does not exist"
Nov 01 01:07:06.400804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.400768    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 01:07:06.405618 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.405594    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-44c9bf-5f5fbdc676-xvlxl"]
Nov 01 01:07:06.513244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:06.508759    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50e39bf-f364-4fc1-b958-9a5a251d0804" path="/var/lib/kubelet/pods/f50e39bf-f364-4fc1-b958-9a5a251d0804/volumes"
Nov 01 01:07:13.249166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:13.236327    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:07:14.241760 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:14.241725    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:07:14.990715 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:14.990670    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:07:22.286829 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.286747    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:22.286829 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.286817    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:22.416244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.416215    4493 generic.go:356] "Generic (PLEG): container finished" podID="d055b277-3003-4362-8457-2852717632b3" containerID="4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00" exitCode=0
Nov 01 01:07:22.416417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.416291    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerDied","Data":"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"}
Nov 01 01:07:22.417422 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.417396    4493 scope.go:122] "RemoveContainer" containerID="4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"
Nov 01 01:07:22.421627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.421574    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:22.421768 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.421743    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:22.421888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:22.421865    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:23.223158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.219659    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:07:23.227154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.225644    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f50e39bf-f364-4fc1-b958-9a5a251d0804" containerName="maas-api"
Nov 01 01:07:23.227154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.225677    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50e39bf-f364-4fc1-b958-9a5a251d0804" containerName="maas-api"
Nov 01 01:07:23.227154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.225815    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50e39bf-f364-4fc1-b958-9a5a251d0804" containerName="maas-api"
Nov 01 01:07:23.239156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.238442    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.291541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.291507    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6b9b61-serving-cert\""
Nov 01 01:07:23.345971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.345939    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:07:23.355979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.355931    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.356209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.356005    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.356209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.356040    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.356209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.356100    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.447843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.447808    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerStarted","Data":"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"}
Nov 01 01:07:23.457429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.457398    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.457580 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.457440    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.457580 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.457502    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.457713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.457694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.458336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.458307    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.460221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.460194    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.460309 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.460274    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.544237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.544116    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\") pod \"maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.554770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.554740    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:23.956376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:23.956346    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:07:23.957837 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:07:23.957801    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f3c424_7472_4070_8fe3_2deb2592241f.slice/crio-0aed875be65cda8f0621dd18ca8f1210da2773cb5c2e73e54dbfcd286dab0fea WatchSource:0}: Error finding container 0aed875be65cda8f0621dd18ca8f1210da2773cb5c2e73e54dbfcd286dab0fea: Status 404 returned error can't find the container with id 0aed875be65cda8f0621dd18ca8f1210da2773cb5c2e73e54dbfcd286dab0fea
Nov 01 01:07:24.233163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:24.231043    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:07:24.233163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:24.231189    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:24.233163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:24.231232    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:24.455075 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:24.455039    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" event={"ID":"d7f3c424-7472-4070-8fe3-2deb2592241f","Type":"ContainerStarted","Data":"0aed875be65cda8f0621dd18ca8f1210da2773cb5c2e73e54dbfcd286dab0fea"}
Nov 01 01:07:25.228219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:25.223521    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:25.228219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:25.223582    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:25.467649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:25.467617    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" event={"ID":"d7f3c424-7472-4070-8fe3-2deb2592241f","Type":"ContainerStarted","Data":"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"}
Nov 01 01:07:25.468071 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:25.467806    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:26.225160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:26.223501    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:26.225160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:26.223565    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:27.236665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:27.236621    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:27.237303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:27.237269    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:27.352455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:27.352385    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" podStartSLOduration=3.437269762 podStartE2EDuration="4.352370121s" totalImagesPullingTime="915.100359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:25.592082797 +0000 UTC m=+2635.269669756" watchObservedRunningTime="2026-11-01 01:07:27.352370121 +0000 UTC m=+2637.029957066"
Nov 01 01:07:27.352915 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:27.352890    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 01:07:27.462995 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:27.462960    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 01:07:28.239157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.237262    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 01:07:28.239157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.237326    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 01:07:28.367996 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.367965    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:07:28.368248 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.368182    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" podUID="f55abc1d-06b4-47d5-b556-996018072110" containerName="maas-api" containerID="cri-o://79cf8112e3058228a72268be5e35a4f85626d01c7aade5585ca94c1ed3635140" gracePeriod=30
Nov 01 01:07:28.556371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.556280    4493 generic.go:356] "Generic (PLEG): container finished" podID="f55abc1d-06b4-47d5-b556-996018072110" containerID="79cf8112e3058228a72268be5e35a4f85626d01c7aade5585ca94c1ed3635140" exitCode=0
Nov 01 01:07:28.556371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.556354    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" event={"ID":"f55abc1d-06b4-47d5-b556-996018072110","Type":"ContainerDied","Data":"79cf8112e3058228a72268be5e35a4f85626d01c7aade5585ca94c1ed3635140"}
Nov 01 01:07:28.720675 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.720644    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:07:28.777339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.777299    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\") pod \"f55abc1d-06b4-47d5-b556-996018072110\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") "
Nov 01 01:07:28.777339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.777346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") pod \"f55abc1d-06b4-47d5-b556-996018072110\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") "
Nov 01 01:07:28.777621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.777439    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\" (UniqueName: \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\") pod \"f55abc1d-06b4-47d5-b556-996018072110\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") "
Nov 01 01:07:28.777621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.777525    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\") pod \"f55abc1d-06b4-47d5-b556-996018072110\" (UID: \"f55abc1d-06b4-47d5-b556-996018072110\") "
Nov 01 01:07:28.777753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.777726    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca" pod "f55abc1d-06b4-47d5-b556-996018072110" (UID: "f55abc1d-06b4-47d5-b556-996018072110"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:28.778099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.778074    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55abc1d-06b4-47d5-b556-996018072110-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:28.779798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.779772    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls" pod "f55abc1d-06b4-47d5-b556-996018072110" (UID: "f55abc1d-06b4-47d5-b556-996018072110"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:28.779911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.779814    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7" pod "f55abc1d-06b4-47d5-b556-996018072110" (UID: "f55abc1d-06b4-47d5-b556-996018072110"). InnerVolumeSpecName "kube-api-access-74bt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:28.779911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.779818    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs" pod "f55abc1d-06b4-47d5-b556-996018072110" (UID: "f55abc1d-06b4-47d5-b556-996018072110"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:28.873345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.873266    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:07:28.874223 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.874199    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f55abc1d-06b4-47d5-b556-996018072110" containerName="maas-api"
Nov 01 01:07:28.874343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.874224    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55abc1d-06b4-47d5-b556-996018072110" containerName="maas-api"
Nov 01 01:07:28.874517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.874499    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55abc1d-06b4-47d5-b556-996018072110" containerName="maas-api"
Nov 01 01:07:28.878591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.878570    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.880196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.879493    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:28.880196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.879524    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74bt7\" (UniqueName: \"kubernetes.io/projected/f55abc1d-06b4-47d5-b556-996018072110-kube-api-access-74bt7\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:28.880196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.879539    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55abc1d-06b4-47d5-b556-996018072110-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:28.932627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.932587    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cbea8a-openshift-default-dockercfg-phptj\""
Nov 01 01:07:28.932802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.932782    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:07:28.980873 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.980837    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.980986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981072    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981125    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981205    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981314 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981284    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981390    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981430    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:28.981510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:28.981493    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwl8\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082508    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082559    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082604    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082623    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082727 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082654    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082763    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.082872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082796    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.082991    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083198 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083176    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083253    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083380 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083357    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwl8\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083444 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083415    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083594 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083572    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.083814 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.083789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.085525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.085505    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.085655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.085641    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.182206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.171208    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwl8\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.182206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.180708    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\") pod \"e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") " pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.217156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.216387    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:29.237159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.234283    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:07:29.237159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.235694    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:07:29.245154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.241379    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:07:29.610199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.610104    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"
Nov 01 01:07:29.615597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.615557    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8" event={"ID":"f55abc1d-06b4-47d5-b556-996018072110","Type":"ContainerDied","Data":"81cde2826aa405f021d3863dd07d42be33d9f7097304ed759d364cce3736397d"}
Nov 01 01:07:29.615773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.615637    4493 scope.go:122] "RemoveContainer" containerID="79cf8112e3058228a72268be5e35a4f85626d01c7aade5585ca94c1ed3635140"
Nov 01 01:07:29.733642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.733619    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:07:29.735346 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:07:29.735314    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d393838_b6da_4012_8830_fe231a36bfc9.slice/crio-95baaca78149404df6c0d0853ce92fc3541c2a5dd3d6d7fa8dd24f3f9c617090 WatchSource:0}: Error finding container 95baaca78149404df6c0d0853ce92fc3541c2a5dd3d6d7fa8dd24f3f9c617090: Status 404 returned error can't find the container with id 95baaca78149404df6c0d0853ce92fc3541c2a5dd3d6d7fa8dd24f3f9c617090
Nov 01 01:07:29.738427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.738397    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:29.738500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.738460    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:29.738500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.738487    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:29.783881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.783850    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:07:29.787451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:29.787429    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-rxrx8"]
Nov 01 01:07:30.380893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.380858    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:07:30.385036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.385018    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407300    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407460 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407540    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407575    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407601    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407740    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407789    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407847 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407816    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.407903 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.407887    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xth\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.437749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.437719    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4143b7-openshift-default-dockercfg-fhj7l\""
Nov 01 01:07:30.447077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.447051    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:07:30.502714 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.502680    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55abc1d-06b4-47d5-b556-996018072110" path="/var/lib/kubelet/pods/f55abc1d-06b4-47d5-b556-996018072110/volumes"
Nov 01 01:07:30.509000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.508975    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509146 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509111    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509210    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509280    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509327    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509380 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509368    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509383    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509488 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509466    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xth\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509527    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509623    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509668    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.509946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.510072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.509997    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.511397 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.511378    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.511807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.511789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.570849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.570820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xth\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.571125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.571101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:30.618024 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.617987    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerStarted","Data":"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"}
Nov 01 01:07:30.618247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.618049    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerStarted","Data":"95baaca78149404df6c0d0853ce92fc3541c2a5dd3d6d7fa8dd24f3f9c617090"}
Nov 01 01:07:30.743351 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.743290    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podStartSLOduration=2.743276106 podStartE2EDuration="2.743276106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:30.740071642 +0000 UTC m=+2640.417658589" watchObservedRunningTime="2026-11-01 01:07:30.743276106 +0000 UTC m=+2640.420863050"
Nov 01 01:07:30.751211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.751182    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4143b7-openshift-default-dockercfg-fhj7l\""
Nov 01 01:07:30.760314 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:30.760289    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:31.196186 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.196154    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:07:31.201763 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.201722    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:31.201914 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.201808    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:31.201914 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.201856    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:31.216944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.216914    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:31.223214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.223190    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:31.520758 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.520728    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:07:31.629286 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.629245    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" event={"ID":"0de359a7-b73d-4760-94f6-e7aaa45844a0","Type":"ContainerStarted","Data":"a372818ebdb4df8e5001f228be0ad2dcd4e4e5f3b51bdefaf5c3699c1ffe0c34"}
Nov 01 01:07:31.629456 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.629304    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" event={"ID":"0de359a7-b73d-4760-94f6-e7aaa45844a0","Type":"ContainerStarted","Data":"f6ee64fa5d4812f1ad7644e896eb344f3a28f9259e7c9195726466bd40b81c14"}
Nov 01 01:07:31.629456 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.629455    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:31.630475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.630453    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:07:31.762831 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.761097    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:31.762831 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:31.761628    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podStartSLOduration=1.76161061 podStartE2EDuration="1.76161061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:31.759798884 +0000 UTC m=+2641.437385831" watchObservedRunningTime="2026-11-01 01:07:31.76161061 +0000 UTC m=+2641.439197556"
Nov 01 01:07:32.353289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:32.353224    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing" containerID="cri-o://bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de" gracePeriod=25
Nov 01 01:07:32.466429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:32.466386    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing" containerID="cri-o://d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b" gracePeriod=25
Nov 01 01:07:32.762737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:32.762705    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:32.763117 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:32.762769    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:33.761293 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:33.761260    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:33.761482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:33.761324    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:34.242160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:34.236226    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:07:34.242160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:34.240221    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:07:34.761401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:34.761357    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:34.761577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:34.761424    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:35.760926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:35.760885    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:35.761313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:35.760944    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:36.761496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:36.761459    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:36.761870 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:36.761511    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:37.760800 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:37.760769    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 01:07:37.760976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:37.760821    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 01:07:38.765725 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:38.765695    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:38.766209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:38.766178    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:38.766815 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:38.766795    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:07:44.226239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:44.226124    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:07:44.226698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:44.226156    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:07:44.989769 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:44.989735    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:07:45.345642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.345558    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:07:45.346073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.345764    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" podUID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" containerName="maas-api" containerID="cri-o://6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c" gracePeriod=30
Nov 01 01:07:45.665385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.665358    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:07:45.681450 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.681418    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\") pod \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") "
Nov 01 01:07:45.681651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.681630    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\" (UniqueName: \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\") pod \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") "
Nov 01 01:07:45.681740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.681712    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") pod \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") "
Nov 01 01:07:45.681798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.681772    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\") pod \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\" (UID: \"edf52fd3-5f28-495b-ac4e-e100fc3bf915\") "
Nov 01 01:07:45.682201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.682173    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca" pod "edf52fd3-5f28-495b-ac4e-e100fc3bf915" (UID: "edf52fd3-5f28-495b-ac4e-e100fc3bf915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:45.683977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.683950    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls" pod "edf52fd3-5f28-495b-ac4e-e100fc3bf915" (UID: "edf52fd3-5f28-495b-ac4e-e100fc3bf915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:45.684399 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.684371    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs" pod "edf52fd3-5f28-495b-ac4e-e100fc3bf915" (UID: "edf52fd3-5f28-495b-ac4e-e100fc3bf915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:45.684600 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.684580    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp" pod "edf52fd3-5f28-495b-ac4e-e100fc3bf915" (UID: "edf52fd3-5f28-495b-ac4e-e100fc3bf915"). InnerVolumeSpecName "kube-api-access-nmsfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:45.783074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.780910    4493 generic.go:356] "Generic (PLEG): container finished" podID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" containerID="6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c" exitCode=0
Nov 01 01:07:45.783074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.780956    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" event={"ID":"edf52fd3-5f28-495b-ac4e-e100fc3bf915","Type":"ContainerDied","Data":"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"}
Nov 01 01:07:45.783074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.781009    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm" event={"ID":"edf52fd3-5f28-495b-ac4e-e100fc3bf915","Type":"ContainerDied","Data":"81e1fb942a4e28be48f3b2b04e2ddedecc1d847af793df213d65d1b82ec83570"}
Nov 01 01:07:45.783074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.781042    4493 scope.go:122] "RemoveContainer" containerID="6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"
Nov 01 01:07:45.783074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.781257    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"
Nov 01 01:07:45.809542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.809444    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmsfp\" (UniqueName: \"kubernetes.io/projected/edf52fd3-5f28-495b-ac4e-e100fc3bf915-kube-api-access-nmsfp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:45.809542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.809480    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:45.809542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.809496    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf52fd3-5f28-495b-ac4e-e100fc3bf915-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:45.809542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.809508    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf52fd3-5f28-495b-ac4e-e100fc3bf915-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:45.853028 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.852931    4493 scope.go:122] "RemoveContainer" containerID="6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"
Nov 01 01:07:45.853380 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:45.853348    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c\": container with ID starting with 6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c not found: ID does not exist" containerID="6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"
Nov 01 01:07:45.853506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.853391    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c"} err="failed to get container status \"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c\": rpc error: code = NotFound desc = could not find container \"6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c\": container with ID starting with 6cd476e07ff602a2bd1c34dd183e5558193a5836fa20d740f26264b5b74cd90c not found: ID does not exist"
Nov 01 01:07:45.993274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.993237    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:07:45.996834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:45.996809    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f73fb0-7d9cfd6c58-ctwtm"]
Nov 01 01:07:46.091754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:46.091726    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:07:46.505051 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:46.505018    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" path="/var/lib/kubelet/pods/edf52fd3-5f28-495b-ac4e-e100fc3bf915/volumes"
Nov 01 01:07:52.681612 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.681587    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 01:07:52.784163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.782763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\" (UniqueName: \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\") pod \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") "
Nov 01 01:07:52.784163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.782937    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\") pod \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\" (UID: \"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b\") "
Nov 01 01:07:52.802661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.802626    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume" pod "4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" (UID: "4bf86aff-5f16-4e51-bf5c-df0e5f75b20b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:52.807870 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.807385    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl" pod "4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" (UID: "4bf86aff-5f16-4e51-bf5c-df0e5f75b20b"). InnerVolumeSpecName "kube-api-access-kqkzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:52.829086 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.829064    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 01:07:52.844524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.844480    4493 generic.go:356] "Generic (PLEG): container finished" podID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerID="d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b" exitCode=0
Nov 01 01:07:52.844717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.844656    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"
Nov 01 01:07:52.844805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.844774    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" event={"ID":"4261e9a3-3a92-4d5b-bd4b-3723857d3302","Type":"ContainerDied","Data":"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"}
Nov 01 01:07:52.844882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.844860    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm" event={"ID":"4261e9a3-3a92-4d5b-bd4b-3723857d3302","Type":"ContainerDied","Data":"4d076aa067089b649e8b84682252972bacf280bc91fa324b93f2c97a2f9d0556"}
Nov 01 01:07:52.844937 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.844892    4493 scope.go:122] "RemoveContainer" containerID="d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"
Nov 01 01:07:52.848061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.848035    4493 generic.go:356] "Generic (PLEG): container finished" podID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerID="bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de" exitCode=0
Nov 01 01:07:52.848252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.848214    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" event={"ID":"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b","Type":"ContainerDied","Data":"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"}
Nov 01 01:07:52.848328 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.848265    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg" event={"ID":"4bf86aff-5f16-4e51-bf5c-df0e5f75b20b","Type":"ContainerDied","Data":"9f4998f7eae8abca804651b314ce4b12958eba422bd02ae1aa828bf38c62b69b"}
Nov 01 01:07:52.848382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.848350    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"
Nov 01 01:07:52.857654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.857581    4493 scope.go:122] "RemoveContainer" containerID="d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"
Nov 01 01:07:52.857897 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:52.857874    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b\": container with ID starting with d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b not found: ID does not exist" containerID="d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"
Nov 01 01:07:52.857982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.857904    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b"} err="failed to get container status \"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b\": rpc error: code = NotFound desc = could not find container \"d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b\": container with ID starting with d6afd2a4b481256ca5450293425d2782cba98d812bcf3f448e6c07e2a73e302b not found: ID does not exist"
Nov 01 01:07:52.857982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.857921    4493 scope.go:122] "RemoveContainer" containerID="bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"
Nov 01 01:07:52.869339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.869244    4493 scope.go:122] "RemoveContainer" containerID="bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"
Nov 01 01:07:52.869668 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:52.869609    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de\": container with ID starting with bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de not found: ID does not exist" containerID="bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"
Nov 01 01:07:52.869668 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.869643    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de"} err="failed to get container status \"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de\": rpc error: code = NotFound desc = could not find container \"bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de\": container with ID starting with bd283d15cf63a09ad006cfa62c2ed760220104236fc0c075be999b1df7ab22de not found: ID does not exist"
Nov 01 01:07:52.884531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.884500    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkzl\" (UniqueName: \"kubernetes.io/projected/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-kube-api-access-kqkzl\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:52.884531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.884529    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:52.985770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.985737    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\" (UniqueName: \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\") pod \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") "
Nov 01 01:07:52.985966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.985941    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\") pod \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\" (UID: \"4261e9a3-3a92-4d5b-bd4b-3723857d3302\") "
Nov 01 01:07:52.986383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.986354    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume" pod "4261e9a3-3a92-4d5b-bd4b-3723857d3302" (UID: "4261e9a3-3a92-4d5b-bd4b-3723857d3302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:52.986673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.986485    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4261e9a3-3a92-4d5b-bd4b-3723857d3302-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:52.988165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:52.988109    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5" pod "4261e9a3-3a92-4d5b-bd4b-3723857d3302" (UID: "4261e9a3-3a92-4d5b-bd4b-3723857d3302"). InnerVolumeSpecName "kube-api-access-848s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:53.087540 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:53.087505    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848s5\" (UniqueName: \"kubernetes.io/projected/4261e9a3-3a92-4d5b-bd4b-3723857d3302-kube-api-access-848s5\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:53.115476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:53.115443    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 01:07:53.121776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:53.121749    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-948e22-7bdf8cd9bd-rkswg"]
Nov 01 01:07:53.266454 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:53.266420    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 01:07:53.273105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:53.273074    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-948e22-55cfd685c9-lx8lm"]
Nov 01 01:07:54.507428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:54.507392    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" path="/var/lib/kubelet/pods/4261e9a3-3a92-4d5b-bd4b-3723857d3302/volumes"
Nov 01 01:07:54.507990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:54.507967    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" path="/var/lib/kubelet/pods/4bf86aff-5f16-4e51-bf5c-df0e5f75b20b/volumes"
Nov 01 01:07:56.903205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.903093    4493 generic.go:356] "Generic (PLEG): container finished" podID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerID="06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272" exitCode=0
Nov 01 01:07:56.903205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.903168    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerDied","Data":"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"}
Nov 01 01:07:56.903658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.903601    4493 scope.go:122] "RemoveContainer" containerID="06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"
Nov 01 01:07:56.905061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.905030    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:56.905199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.905115    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:56.905278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:56.905203    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:07:57.269703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.269667    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 01:07:57.269904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.269879    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerName="maas-api" containerID="cri-o://719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0" gracePeriod=30
Nov 01 01:07:57.348945 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.348906    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": dial tcp 10.133.0.47:8443: connect: connection refused"
Nov 01 01:07:57.669053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.669029    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 01:07:57.844001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.843947    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\") pod \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") "
Nov 01 01:07:57.844240 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.844052    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\" (UniqueName: \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\") pod \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") "
Nov 01 01:07:57.844333 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.844239    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") pod \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") "
Nov 01 01:07:57.844333 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.844291    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\") pod \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\" (UID: \"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992\") "
Nov 01 01:07:57.847262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.846240    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca" pod "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" (UID: "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:07:57.848834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.848806    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z" pod "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" (UID: "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992"). InnerVolumeSpecName "kube-api-access-7f82z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:07:57.848967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.848882    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs" pod "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" (UID: "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:57.849578 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.849536    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls" pod "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" (UID: "c02cb64a-b4ab-4b3b-ab40-9f7ac7413992"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:07:57.912993 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.912900    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerStarted","Data":"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"}
Nov 01 01:07:57.915114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.915072    4493 generic.go:356] "Generic (PLEG): container finished" podID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerID="719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0" exitCode=0
Nov 01 01:07:57.915291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.915166    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" event={"ID":"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992","Type":"ContainerDied","Data":"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"}
Nov 01 01:07:57.915291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.915207    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"
Nov 01 01:07:57.915291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.915228    4493 scope.go:122] "RemoveContainer" containerID="719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"
Nov 01 01:07:57.941040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.915216    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6" event={"ID":"c02cb64a-b4ab-4b3b-ab40-9f7ac7413992","Type":"ContainerDied","Data":"514634266bdd10d4754929c9a3d6dd24d10864fdac31ddd86eed18c89dd64093"}
Nov 01 01:07:57.946265 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.946235    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:57.946404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.946270    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f82z\" (UniqueName: \"kubernetes.io/projected/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-kube-api-access-7f82z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:57.946404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.946291    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:57.946404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.946304    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:07:57.952318 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.952294    4493 scope.go:122] "RemoveContainer" containerID="719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"
Nov 01 01:07:57.952693 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:07:57.952670    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0\": container with ID starting with 719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0 not found: ID does not exist" containerID="719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"
Nov 01 01:07:57.952770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:57.952702    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0"} err="failed to get container status \"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0\": rpc error: code = NotFound desc = could not find container \"719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0\": container with ID starting with 719adaa215ad0b971f7905d74f8c53d1959ef5fb6e1a25b6541f7bea3d3c1ec0 not found: ID does not exist"
Nov 01 01:07:58.160260 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.160183    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 01:07:58.220213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.217090    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4143b7-85b648bbf4-mwpb6"]
Nov 01 01:07:58.246160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.245896    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:07:58.246160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.246037    4493 patch_prober.go:29] interesting pod/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 01:07:58.246160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.246099    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 01:07:58.505405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:58.505322    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" path="/var/lib/kubelet/pods/c02cb64a-b4ab-4b3b-ab40-9f7ac7413992/volumes"
Nov 01 01:07:59.244231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:59.244195    4493 patch_prober.go:29] interesting pod/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 01:07:59.244628 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:07:59.244263    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 01:08:00.244678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:00.244642    4493 patch_prober.go:29] interesting pod/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 01:08:00.245191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:00.244704    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 01:08:00.353858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:00.353824    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 01:08:01.244119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:01.244079    4493 patch_prober.go:29] interesting pod/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 01:08:01.244326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:01.244154    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 01:08:02.244756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:02.244716    4493 patch_prober.go:29] interesting pod/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 01:08:02.245209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:02.244795    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 01:08:02.800457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:02.800419    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:08:03.251053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:03.251026    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:08:03.252258 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:03.252235    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:08:06.880495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:06.880450    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:08:06.881020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:06.880734    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" containerID="cri-o://a372818ebdb4df8e5001f228be0ad2dcd4e4e5f3b51bdefaf5c3699c1ffe0c34" gracePeriod=30
Nov 01 01:08:08.768182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:08.768118    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:08:08.768599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:08.768201    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:08:12.042107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.042074    4493 generic.go:356] "Generic (PLEG): container finished" podID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerID="a372818ebdb4df8e5001f228be0ad2dcd4e4e5f3b51bdefaf5c3699c1ffe0c34" exitCode=0
Nov 01 01:08:12.042487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.042168    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" event={"ID":"0de359a7-b73d-4760-94f6-e7aaa45844a0","Type":"ContainerDied","Data":"a372818ebdb4df8e5001f228be0ad2dcd4e4e5f3b51bdefaf5c3699c1ffe0c34"}
Nov 01 01:08:12.142846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.142775    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:08:12.285566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285535    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.285566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285578    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.285818 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285632    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.285818 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285684    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.285911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285852    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:12.285911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285855    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.286020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285984    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.286020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.285976    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:12.286158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.286078    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:12.286222 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.286178    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:08:12.286326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.286303    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.286408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.286351    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.286408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.286373    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\") pod \"0de359a7-b73d-4760-94f6-e7aaa45844a0\" (UID: \"0de359a7-b73d-4760-94f6-e7aaa45844a0\") "
Nov 01 01:08:12.287026 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.287003    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.287413 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.287389    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.287539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.287415    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de359a7-b73d-4760-94f6-e7aaa45844a0-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.287539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.287430    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.287539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.287020    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:12.288536 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.288507    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:08:12.288715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.288694    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:08:12.288786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.288723    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "kube-api-access-76xth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:08:12.288852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.288837    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy" pod "0de359a7-b73d-4760-94f6-e7aaa45844a0" (UID: "0de359a7-b73d-4760-94f6-e7aaa45844a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:12.389147 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.389107    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.389336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.389176    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.389336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.389188    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.389336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.389198    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xth\" (UniqueName: \"kubernetes.io/projected/0de359a7-b73d-4760-94f6-e7aaa45844a0-kube-api-access-76xth\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:12.389336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:12.389209    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de359a7-b73d-4760-94f6-e7aaa45844a0-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:13.049467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:13.049434    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"
Nov 01 01:08:13.049878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:13.049465    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb" event={"ID":"0de359a7-b73d-4760-94f6-e7aaa45844a0","Type":"ContainerDied","Data":"f6ee64fa5d4812f1ad7644e896eb344f3a28f9259e7c9195726466bd40b81c14"}
Nov 01 01:08:13.049878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:13.049529    4493 scope.go:122] "RemoveContainer" containerID="a372818ebdb4df8e5001f228be0ad2dcd4e4e5f3b51bdefaf5c3699c1ffe0c34"
Nov 01 01:08:13.236576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:13.232364    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:08:13.275503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:13.275473    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-kbqkb"]
Nov 01 01:08:14.503918 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:14.503890    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" path="/var/lib/kubelet/pods/0de359a7-b73d-4760-94f6-e7aaa45844a0/volumes"
Nov 01 01:08:14.990322 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:08:14.990286    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:08:15.381693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.381572    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" containerID="cri-o://0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47" gracePeriod=45
Nov 01 01:08:15.427006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.426973    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Nov 01 01:08:15.663035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.663003    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 01:08:15.736292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736261    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736318    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\" (UniqueName: \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736435    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736509    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736563    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736647    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\") pod \"e8d23648-a719-40e5-873b-a05e69faddad\" (UID: \"e8d23648-a719-40e5-873b-a05e69faddad\") "
Nov 01 01:08:15.736703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736661    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:15.736812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736716    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:15.736897 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.736873    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:15.737092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.737075    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:15.737175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.737095    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:15.737175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.737104    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:15.738687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.738666    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:08:15.738788 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.738775    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "kube-api-access-slz9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:08:15.738850 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.738837    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm" pod "e8d23648-a719-40e5-873b-a05e69faddad" (UID: "e8d23648-a719-40e5-873b-a05e69faddad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:08:15.838648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.838612    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slz9g\" (UniqueName: \"kubernetes.io/projected/e8d23648-a719-40e5-873b-a05e69faddad-kube-api-access-slz9g\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:15.838648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.838646    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d23648-a719-40e5-873b-a05e69faddad-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:15.838648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:15.838656    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d23648-a719-40e5-873b-a05e69faddad-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:16.070033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.070000    4493 generic.go:356] "Generic (PLEG): container finished" podID="e8d23648-a719-40e5-873b-a05e69faddad" containerID="0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47" exitCode=0
Nov 01 01:08:16.070231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.070045    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"
Nov 01 01:08:16.070231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.070088    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" event={"ID":"e8d23648-a719-40e5-873b-a05e69faddad","Type":"ContainerDied","Data":"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"}
Nov 01 01:08:16.070231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.070146    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d" event={"ID":"e8d23648-a719-40e5-873b-a05e69faddad","Type":"ContainerDied","Data":"c818fa1035fd9dc271ec86eb206fd593f9c3cc5068657d766e9d1c6d2eea9bf5"}
Nov 01 01:08:16.070231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.070169    4493 scope.go:122] "RemoveContainer" containerID="0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"
Nov 01 01:08:16.080722 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.080704    4493 scope.go:122] "RemoveContainer" containerID="0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"
Nov 01 01:08:16.081005 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:08:16.080982    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47\": container with ID starting with 0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47 not found: ID does not exist" containerID="0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"
Nov 01 01:08:16.081057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.081016    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47"} err="failed to get container status \"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47\": rpc error: code = NotFound desc = could not find container \"0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47\": container with ID starting with 0eea801b975abb04ba392bd071776629822120a8dc85348e334e03bf0e795e47 not found: ID does not exist"
Nov 01 01:08:16.234683 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.234650    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 01:08:16.238699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.238674    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/premium-w0-32439f-kserve-674495c4c5-7sc2d"]
Nov 01 01:08:16.503881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:16.503850    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d23648-a719-40e5-873b-a05e69faddad" path="/var/lib/kubelet/pods/e8d23648-a719-40e5-873b-a05e69faddad/volumes"
Nov 01 01:08:19.469001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.468959    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 01:08:19.469549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.469242    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" podUID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" containerName="maas-api" containerID="cri-o://7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03" gracePeriod=30
Nov 01 01:08:19.818486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.818460    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 01:08:19.958976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.958945    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\" (UniqueName: \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\") pod \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") "
Nov 01 01:08:19.959179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.959027    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") pod \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") "
Nov 01 01:08:19.959179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.959112    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\") pod \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") "
Nov 01 01:08:19.959310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.959292    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\") pod \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\" (UID: \"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1\") "
Nov 01 01:08:19.959985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.959960    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca" pod "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" (UID: "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:08:19.962003 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.961977    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls" pod "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" (UID: "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:08:19.970608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.970576    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs" pod "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" (UID: "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:08:19.970724 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:19.970608    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666" pod "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" (UID: "6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1"). InnerVolumeSpecName "kube-api-access-b4666". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:08:20.061195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.061085    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:20.061195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.061116    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4666\" (UniqueName: \"kubernetes.io/projected/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-kube-api-access-b4666\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:20.061195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.061160    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:20.061195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.061170    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:08:20.095567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.095536    4493 generic.go:356] "Generic (PLEG): container finished" podID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" containerID="7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03" exitCode=0
Nov 01 01:08:20.095744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.095590    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"
Nov 01 01:08:20.095744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.095616    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" event={"ID":"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1","Type":"ContainerDied","Data":"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"}
Nov 01 01:08:20.095744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.095712    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g" event={"ID":"6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1","Type":"ContainerDied","Data":"2ba30c4d233767476e07666d83589eb875d3984032cbd26b85e8e5ea9001c137"}
Nov 01 01:08:20.095744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.095740    4493 scope.go:122] "RemoveContainer" containerID="7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"
Nov 01 01:08:20.106145 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.106111    4493 scope.go:122] "RemoveContainer" containerID="7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"
Nov 01 01:08:20.106398 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:08:20.106378    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03\": container with ID starting with 7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03 not found: ID does not exist" containerID="7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"
Nov 01 01:08:20.106463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.106407    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03"} err="failed to get container status \"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03\": rpc error: code = NotFound desc = could not find container \"7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03\": container with ID starting with 7f5c8eb21ff7a0cadd681e28e899fbdf1c15abf74fe837e7b030452246356d03 not found: ID does not exist"
Nov 01 01:08:20.260636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.260604    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 01:08:20.266589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.266558    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-948e22-59865d5c7d-4882g"]
Nov 01 01:08:20.502548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:20.502515    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" path="/var/lib/kubelet/pods/6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1/volumes"
Nov 01 01:08:23.685510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.685474    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:08:23.686099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686083    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686100    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686128    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" containerName="maas-api"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686148    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" containerName="maas-api"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686157    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerName="maas-api"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686161    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerName="maas-api"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686167    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686172    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686180    4493 cpu_manager.go:414] "containerMap: removing container" podUID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" containerName="maas-api"
Nov 01 01:08:23.686181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686185    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" containerName="maas-api"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686209    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686214    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686228    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686233    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686318    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6e4db4-7e1b-4837-8fac-6c27f7e99fe1" containerName="maas-api"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686328    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02cb64a-b4ab-4b3b-ab40-9f7ac7413992" containerName="maas-api"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686334    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d23648-a719-40e5-873b-a05e69faddad" containerName="main"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686340    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf52fd3-5f28-495b-ac4e-e100fc3bf915" containerName="maas-api"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686348    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de359a7-b73d-4760-94f6-e7aaa45844a0" containerName="istio-proxy"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686354    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf86aff-5f16-4e51-bf5c-df0e5f75b20b" containerName="payload-processing"
Nov 01 01:08:23.686496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.686360    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4261e9a3-3a92-4d5b-bd4b-3723857d3302" containerName="payload-pre-processing"
Nov 01 01:08:23.691177 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.691159    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743602    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743654    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743720    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2sv\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743796    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743870    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743913    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743955    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.743986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.744023    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.755164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.744484    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1402e8-openshift-default-dockercfg-pzwd4\""
Nov 01 01:08:23.866973 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.866928    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.867378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.867350    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.867540 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.867522    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.867671 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.867656    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.867780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.867764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.867894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.867879    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.868103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.868086    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.868255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.868239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.868392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.868380    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2sv\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.877156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.873512    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.877358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.874550    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.877464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.875185    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.877556 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.876148    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.889499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.876379    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.910157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.903652    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.910157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.904179    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.910157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.907424    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:08:23.934318 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.934288    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2sv\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:23.936968 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:23.936908    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\") pod \"e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") " pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:24.005189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:24.005125    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:24.435485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:24.435461    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:08:24.437342 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:08:24.437317    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f72f912_d6b2_41d9_a4b1_64c582020a36.slice/crio-d2b69debcfb6ff63e1b3ef440614b09cf6936709af38ae557592820910583c57 WatchSource:0}: Error finding container d2b69debcfb6ff63e1b3ef440614b09cf6936709af38ae557592820910583c57: Status 404 returned error can't find the container with id d2b69debcfb6ff63e1b3ef440614b09cf6936709af38ae557592820910583c57
Nov 01 01:08:24.439868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:24.439834    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:24.439950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:24.439899    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:24.439950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:24.439926    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:25.150019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:25.149983    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerStarted","Data":"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"}
Nov 01 01:08:25.150487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:25.150036    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerStarted","Data":"d2b69debcfb6ff63e1b3ef440614b09cf6936709af38ae557592820910583c57"}
Nov 01 01:08:25.271447 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:25.271396    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podStartSLOduration=2.271380792 podStartE2EDuration="2.271380792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:25.269488922 +0000 UTC m=+2694.947075869" watchObservedRunningTime="2026-11-01 01:08:25.271380792 +0000 UTC m=+2694.948967814"
Nov 01 01:08:26.006319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:26.006209    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:26.011876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:26.011851    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:26.156368 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:26.156343    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:26.157355 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:26.157339    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:08:31.923153 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:31.918892    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:08:31.936161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:31.928943    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.032317 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.032285    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1402e8\""
Nov 01 01:08:32.085379 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.085344    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:08:32.087789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.087768    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:08:32.089537 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.089511    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.089792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.089777    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qpj\" (UniqueName: \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.091577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.091557    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.147658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.147623    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:08:32.150735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.150706    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1402e8-dockercfg-kfdf8\""
Nov 01 01:08:32.190952 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.190879    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kr4g\" (UniqueName: \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.191110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.190948    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.191191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.191159    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qpj\" (UniqueName: \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.191249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.191220    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.191997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.191974    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.252537 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.252506    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qpj\" (UniqueName: \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\") pod \"payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.257221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.257202    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:32.292969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.292931    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kr4g\" (UniqueName: \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.293147 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.293039    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.293759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.293741    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.356713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.356686    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kr4g\" (UniqueName: \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\") pod \"payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.404744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.404714    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:32.657224 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.657200    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:08:32.659117 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:08:32.659085    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5eaba8_d92f_4826_82c0_2861f4bcf88a.slice/crio-b6e774049ee3c6e1f6443981ed3c0331bdfedb436cd799260592ae4c71ad3c2a WatchSource:0}: Error finding container b6e774049ee3c6e1f6443981ed3c0331bdfedb436cd799260592ae4c71ad3c2a: Status 404 returned error can't find the container with id b6e774049ee3c6e1f6443981ed3c0331bdfedb436cd799260592ae4c71ad3c2a
Nov 01 01:08:32.784001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:32.783979    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:08:32.786016 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:08:32.785982    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0633dc74_e95f_47c5_ae58_16f58206db1f.slice/crio-2a53b5245ae55ffd6abf88cd99c0ac6169db6546515403e4d43d7f2d99357ba0 WatchSource:0}: Error finding container 2a53b5245ae55ffd6abf88cd99c0ac6169db6546515403e4d43d7f2d99357ba0: Status 404 returned error can't find the container with id 2a53b5245ae55ffd6abf88cd99c0ac6169db6546515403e4d43d7f2d99357ba0
Nov 01 01:08:33.200605 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.200566    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" event={"ID":"0633dc74-e95f-47c5-ae58-16f58206db1f","Type":"ContainerStarted","Data":"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"}
Nov 01 01:08:33.201084 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.200627    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" event={"ID":"0633dc74-e95f-47c5-ae58-16f58206db1f","Type":"ContainerStarted","Data":"2a53b5245ae55ffd6abf88cd99c0ac6169db6546515403e4d43d7f2d99357ba0"}
Nov 01 01:08:33.201084 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.200648    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:33.202367 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.202339    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" event={"ID":"fb5eaba8-d92f-4826-82c0-2861f4bcf88a","Type":"ContainerStarted","Data":"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"}
Nov 01 01:08:33.202479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.202383    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" event={"ID":"fb5eaba8-d92f-4826-82c0-2861f4bcf88a","Type":"ContainerStarted","Data":"b6e774049ee3c6e1f6443981ed3c0331bdfedb436cd799260592ae4c71ad3c2a"}
Nov 01 01:08:33.202479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.202438    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:33.317897 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:33.317849    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" podStartSLOduration=2.317834379 podStartE2EDuration="2.317834379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:33.31601631 +0000 UTC m=+2702.993603255" watchObservedRunningTime="2026-11-01 01:08:33.317834379 +0000 UTC m=+2702.995421356"
Nov 01 01:08:36.764568 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:36.764504    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" podStartSLOduration=5.76448054 podStartE2EDuration="5.76448054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:33.441265942 +0000 UTC m=+2703.118852888" watchObservedRunningTime="2026-11-01 01:08:36.76448054 +0000 UTC m=+2706.442067485"
Nov 01 01:08:36.766615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:36.766581    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:08:44.211854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:44.211824    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:08:44.212372 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:44.211905    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:08:44.990218 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:08:44.990177    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:08:47.301225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.301196    4493 generic.go:356] "Generic (PLEG): container finished" podID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerID="d40ab96fe59ebe104d1023e7b5735c7471e13e0b9b31b372c3bbd8c1fc650026" exitCode=0
Nov 01 01:08:47.301684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.301274    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerDied","Data":"d40ab96fe59ebe104d1023e7b5735c7471e13e0b9b31b372c3bbd8c1fc650026"}
Nov 01 01:08:47.301731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.301690    4493 scope.go:122] "RemoveContainer" containerID="d40ab96fe59ebe104d1023e7b5735c7471e13e0b9b31b372c3bbd8c1fc650026"
Nov 01 01:08:47.303170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.303122    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:47.303273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.303231    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:47.303273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:47.303266    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:08:48.309577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:48.309535    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerStarted","Data":"a2e862f656f62ca12a0621e4a345d4ad2401d65e00ba5d052dc1a6b2f9614b09"}
Nov 01 01:08:49.040346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:49.040307    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:08:49.040558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:49.040520    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 01:08:49.040620 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:49.040577    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 01:08:50.040161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:50.036564    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 01:08:50.040161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:50.036642    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 01:08:51.040592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:51.040550    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 01:08:51.041033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:51.040627    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 01:08:52.033511 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:52.033471    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 01:08:52.033812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:52.033783    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 01:08:53.040798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:53.040760    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 01:08:53.041243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:53.040825    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 01:08:54.036532 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:54.036494    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:08:54.036879 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:54.036857    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:08:54.037619 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:08:54.037603    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:09:00.466485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.466448    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:00.470662 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.470637    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.523360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.523320    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-32439f-openshift-default-dockercfg-n9rr6\""
Nov 01 01:09:00.576531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.576490    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:00.636439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636404    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636452    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636560    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636629    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636720    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636762    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbcn\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.636966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636943    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.637025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.636984    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.637025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.637006    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738564 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738522    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738600    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738647    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738678    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.738750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738713    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739009 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738751    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbcn\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739009 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738866    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739009 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738905    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739009 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.738996    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.739357    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.739470    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.739649    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.739748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.739688    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.741108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.741088    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.741779 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.741763    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.800175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.800100    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:00.800330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:00.800207    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbcn\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:01.085219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.085114    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:01.518038 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.518018    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:01.519697 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:09:01.519670    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fe9570_571e_4a1e_b954_40f64a70a519.slice/crio-dcb9eab4db490daa78b490862c0c6b07332e2dd0f3d5f99d43d37a5686a0f590 WatchSource:0}: Error finding container dcb9eab4db490daa78b490862c0c6b07332e2dd0f3d5f99d43d37a5686a0f590: Status 404 returned error can't find the container with id dcb9eab4db490daa78b490862c0c6b07332e2dd0f3d5f99d43d37a5686a0f590
Nov 01 01:09:01.522272 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.522242    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:01.522329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.522315    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:01.522369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.522341    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:01.605969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.605931    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:09:01.606296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.606263    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" containerID="cri-o://09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b" gracePeriod=30
Nov 01 01:09:01.619606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.619577    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 01:09:01.620022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.619990    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy" containerID="cri-o://7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07" gracePeriod=30
Nov 01 01:09:01.634553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.634523    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:09:01.634780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.634754    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy" containerID="cri-o://44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148" gracePeriod=30
Nov 01 01:09:01.637670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.637645    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:09:01.819848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.819763    4493 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:09:01.819848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:01.819825    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:09:02.440210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:02.437550    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy" containerID="cri-o://b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666" gracePeriod=30
Nov 01 01:09:02.440210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:02.437747    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" event={"ID":"20fe9570-571e-4a1e-b954-40f64a70a519","Type":"ContainerStarted","Data":"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"}
Nov 01 01:09:02.440210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:02.437864    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" event={"ID":"20fe9570-571e-4a1e-b954-40f64a70a519","Type":"ContainerStarted","Data":"dcb9eab4db490daa78b490862c0c6b07332e2dd0f3d5f99d43d37a5686a0f590"}
Nov 01 01:09:03.085421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:03.085389    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:03.087097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:03.087075    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:03.087249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:03.087127    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:04.085462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:04.085428    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:04.086039 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:04.085477    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:05.086451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:05.086408    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:05.086853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:05.086497    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:06.086296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.086262    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:06.086482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.086314    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:06.928667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.928346    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:09:06.948426 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948395    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948463    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948545 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948504    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948545    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948616    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948680    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948754    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948898    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.948946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.948930    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\") pod \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\" (UID: \"11fc6cd1-3a42-4283-9e71-9540c51e37b6\") "
Nov 01 01:09:06.949440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.949413    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:06.949568 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.949541    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:06.949785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.949767    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:06.949966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.949950    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:06.950075 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.950050    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:06.950630 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.950609    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:06.953048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.953027    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "kube-api-access-kp6hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:06.953383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.953234    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:06.957345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.957317    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:06.957510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.957492    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 01:09:06.957674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:06.957656    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo" pod "11fc6cd1-3a42-4283-9e71-9540c51e37b6" (UID: "11fc6cd1-3a42-4283-9e71-9540c51e37b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:09:07.005434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.005357    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:09:07.050300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050271    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.050300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050310    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050329    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050349    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050413    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050429    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050461    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.050577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050533    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.050943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050720    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051008 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050948    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051008 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.050980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:07.051121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051076    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051091    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051121 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051110    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051148    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051198    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051220    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051228    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051265    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.051316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051298    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051315    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051333    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\") pod \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\" (UID: \"28b92ba3-60ed-4a0e-8067-7daf614d6e7c\") "
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051431    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051436    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\") pod \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\" (UID: \"9b8171f6-7f06-4ce7-9d0b-171ab8721445\") "
Nov 01 01:09:07.051638 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.051632    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.052116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052039    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052066    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052081    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052093    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052107    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052120    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052150    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052163    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052175    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052189    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052231    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052246    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052262    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052276    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6hp\" (UniqueName: \"kubernetes.io/projected/11fc6cd1-3a42-4283-9e71-9540c51e37b6-kube-api-access-kp6hp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052292    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052306    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052320    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc6cd1-3a42-4283-9e71-9540c51e37b6-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.052405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.052377    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:07.053325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.053304    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "kube-api-access-2vs4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.054014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.053996    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.054081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.054021    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:09:07.054854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.054813    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "kube-api-access-vxxv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.054928 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.054903    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.055231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.055211    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.055455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.055440    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy" pod "9b8171f6-7f06-4ce7-9d0b-171ab8721445" (UID: "9b8171f6-7f06-4ce7-9d0b-171ab8721445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.055601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.055578    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo" pod "28b92ba3-60ed-4a0e-8067-7daf614d6e7c" (UID: "28b92ba3-60ed-4a0e-8067-7daf614d6e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:09:07.086278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.085568    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:07.086278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.085631    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:07.110044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.109983    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podStartSLOduration=7.10996519 podStartE2EDuration="7.10996519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:02.724636766 +0000 UTC m=+2732.402223738" watchObservedRunningTime="2026-11-01 01:09:07.10996519 +0000 UTC m=+2736.787552134"
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153210    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxv9\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-kube-api-access-vxxv9\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153243    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153258    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153270    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153281    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153295    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153307    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8171f6-7f06-4ce7-9d0b-171ab8721445-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153320    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vs4g\" (UniqueName: \"kubernetes.io/projected/9b8171f6-7f06-4ce7-9d0b-171ab8721445-kube-api-access-2vs4g\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.153616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.153331    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b92ba3-60ed-4a0e-8067-7daf614d6e7c-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.475440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.475407    4493 generic.go:356] "Generic (PLEG): container finished" podID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerID="7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07" exitCode=0
Nov 01 01:09:07.475622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.475467    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"
Nov 01 01:09:07.475702 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.475617    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" event={"ID":"9b8171f6-7f06-4ce7-9d0b-171ab8721445","Type":"ContainerDied","Data":"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"}
Nov 01 01:09:07.475702 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.475673    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5" event={"ID":"9b8171f6-7f06-4ce7-9d0b-171ab8721445","Type":"ContainerDied","Data":"394cb115cc833013b99718e810a45ac2338514b3ac442049ba21c37f9d2a9fa3"}
Nov 01 01:09:07.475807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.475708    4493 scope.go:122] "RemoveContainer" containerID="7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"
Nov 01 01:09:07.485165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.482103    4493 generic.go:356] "Generic (PLEG): container finished" podID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerID="44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148" exitCode=0
Nov 01 01:09:07.485165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.482179    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"
Nov 01 01:09:07.485165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.482239    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" event={"ID":"11fc6cd1-3a42-4283-9e71-9540c51e37b6","Type":"ContainerDied","Data":"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"}
Nov 01 01:09:07.485165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.482323    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj" event={"ID":"11fc6cd1-3a42-4283-9e71-9540c51e37b6","Type":"ContainerDied","Data":"3b8630607e841201d99cb4c94c2eaa9d8c5005d456c2c5eadf51dbb13847e533"}
Nov 01 01:09:07.485499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.485205    4493 generic.go:356] "Generic (PLEG): container finished" podID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerID="09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b" exitCode=0
Nov 01 01:09:07.485499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.485260    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" event={"ID":"28b92ba3-60ed-4a0e-8067-7daf614d6e7c","Type":"ContainerDied","Data":"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"}
Nov 01 01:09:07.485499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.485313    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh" event={"ID":"28b92ba3-60ed-4a0e-8067-7daf614d6e7c","Type":"ContainerDied","Data":"e9065cd7337775f4b2f040d7480339711ffebff8a9d254f023dbea8572508a27"}
Nov 01 01:09:07.485499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.485383    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"
Nov 01 01:09:07.556106 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.556085    4493 scope.go:122] "RemoveContainer" containerID="7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"
Nov 01 01:09:07.556455 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:07.556435    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07\": container with ID starting with 7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07 not found: ID does not exist" containerID="7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"
Nov 01 01:09:07.556548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.556463    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07"} err="failed to get container status \"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07\": rpc error: code = NotFound desc = could not find container \"7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07\": container with ID starting with 7bfd2bf4ecde52e6d695f4d4aa11c1295e5e558761e38f3a9a838ceafb818a07 not found: ID does not exist"
Nov 01 01:09:07.556548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.556483    4493 scope.go:122] "RemoveContainer" containerID="44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"
Nov 01 01:09:07.567081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.566976    4493 scope.go:122] "RemoveContainer" containerID="44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"
Nov 01 01:09:07.567349 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:07.567307    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148\": container with ID starting with 44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148 not found: ID does not exist" containerID="44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"
Nov 01 01:09:07.567516 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.567491    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148"} err="failed to get container status \"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148\": rpc error: code = NotFound desc = could not find container \"44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148\": container with ID starting with 44ba3ac3f914eafecae0bf4b9354ce3024bc3d31efc4a7ff779aee8ebe0c2148 not found: ID does not exist"
Nov 01 01:09:07.567516 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.567518    4493 scope.go:122] "RemoveContainer" containerID="09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"
Nov 01 01:09:07.582254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.582224    4493 scope.go:122] "RemoveContainer" containerID="09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"
Nov 01 01:09:07.582701 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:07.582669    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b\": container with ID starting with 09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b not found: ID does not exist" containerID="09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"
Nov 01 01:09:07.582810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.582703    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b"} err="failed to get container status \"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b\": rpc error: code = NotFound desc = could not find container \"09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b\": container with ID starting with 09224e2a3b0d1ea96473108b349065075033e270c9f0ad295b45d57f5763c40b not found: ID does not exist"
Nov 01 01:09:07.666962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.666937    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 01:09:07.667163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.667128    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-vnpt5"]
Nov 01 01:09:07.743825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.743801    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:09:07.764345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764316    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764358    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764490    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764544    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764613    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764645    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764674    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764795    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\") pod \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\" (UID: \"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b\") "
Nov 01 01:09:07.764957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764820    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.764957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.764891    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:07.765068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765003    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.765250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765202    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.765331 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765270    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.765807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765785    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.765920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765813    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.765920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765832    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.765920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765845    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.765920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.765859    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.767170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.767111    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:09:07.767446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.767418    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "kube-api-access-j4z4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.767585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.767561    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:07.767650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.767629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token" pod "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" (UID: "6aba9d9f-09d2-4607-9b1c-f8bf4c67408b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:07.775177 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.775068    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:09:07.781250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.781224    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-45kwh"]
Nov 01 01:09:07.866901 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.866873    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.866901 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.866900    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.867097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.866913    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4z4z\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-kube-api-access-j4z4z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.867097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.866922    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:07.894455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.894382    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:09:07.897661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:07.897632    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-rx9pj"]
Nov 01 01:09:08.086426 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.086392    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 01 01:09:08.086644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.086447    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 01 01:09:08.492860 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.492826    4493 generic.go:356] "Generic (PLEG): container finished" podID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerID="b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666" exitCode=0
Nov 01 01:09:08.493383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.492912    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"
Nov 01 01:09:08.493383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.492915    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" event={"ID":"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b","Type":"ContainerDied","Data":"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"}
Nov 01 01:09:08.493383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.493018    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll" event={"ID":"6aba9d9f-09d2-4607-9b1c-f8bf4c67408b","Type":"ContainerDied","Data":"bbe5930741af1de73c42a485497ce6f6849d7afab909269cfb55788bd825c5ef"}
Nov 01 01:09:08.493383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.493047    4493 scope.go:122] "RemoveContainer" containerID="b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"
Nov 01 01:09:08.504456 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.504435    4493 scope.go:122] "RemoveContainer" containerID="b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"
Nov 01 01:09:08.504775 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:08.504758    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666\": container with ID starting with b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666 not found: ID does not exist" containerID="b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"
Nov 01 01:09:08.504825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.504784    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666"} err="failed to get container status \"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666\": rpc error: code = NotFound desc = could not find container \"b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666\": container with ID starting with b69dbe6e59c594c3a7c0d09a066a9272452e9319a662332efc8a0b3cafc87666 not found: ID does not exist"
Nov 01 01:09:08.504967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.504950    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" path="/var/lib/kubelet/pods/11fc6cd1-3a42-4283-9e71-9540c51e37b6/volumes"
Nov 01 01:09:08.505428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.505415    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" path="/var/lib/kubelet/pods/28b92ba3-60ed-4a0e-8067-7daf614d6e7c/volumes"
Nov 01 01:09:08.505829 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.505817    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" path="/var/lib/kubelet/pods/9b8171f6-7f06-4ce7-9d0b-171ab8721445/volumes"
Nov 01 01:09:08.662281 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.662227    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:09:08.667375 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:08.667346    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-fxzll"]
Nov 01 01:09:09.134259 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:09.134228    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:09.134648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:09.134619    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:09.136624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:09.136602    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:10.503126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:10.503093    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" path="/var/lib/kubelet/pods/6aba9d9f-09d2-4607-9b1c-f8bf4c67408b/volumes"
Nov 01 01:09:14.989082 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:14.989001    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:09:17.168610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:17.168576    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:17.169033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:17.168779    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy" containerID="cri-o://0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131" gracePeriod=30
Nov 01 01:09:19.789418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:19.789385    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:09:22.493232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.493201    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:22.576967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.576937    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577013    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577078    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577190    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577238    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577291    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577564 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577442    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577564 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577462    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:22.577564 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577499    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\") pod \"20fe9570-571e-4a1e-b954-40f64a70a519\" (UID: \"20fe9570-571e-4a1e-b954-40f64a70a519\") "
Nov 01 01:09:22.577849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577816    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:22.578001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577892    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:22.578078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578026    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:22.578078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.577967    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:22.578233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578166    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.578233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578189    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.578233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578204    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.578233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578217    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.578233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.578230    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fe9570-571e-4a1e-b954-40f64a70a519-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.580047 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.580007    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "kube-api-access-7nbcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:22.580199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.580157    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:09:22.580483 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.580459    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:22.581473 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.581447    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy" pod "20fe9570-571e-4a1e-b954-40f64a70a519" (UID: "20fe9570-571e-4a1e-b954-40f64a70a519"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:09:22.583101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.583077    4493 generic.go:356] "Generic (PLEG): container finished" podID="20fe9570-571e-4a1e-b954-40f64a70a519" containerID="0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131" exitCode=0
Nov 01 01:09:22.583211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.583118    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" event={"ID":"20fe9570-571e-4a1e-b954-40f64a70a519","Type":"ContainerDied","Data":"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"}
Nov 01 01:09:22.583263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.583218    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8" event={"ID":"20fe9570-571e-4a1e-b954-40f64a70a519","Type":"ContainerDied","Data":"dcb9eab4db490daa78b490862c0c6b07332e2dd0f3d5f99d43d37a5686a0f590"}
Nov 01 01:09:22.583263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.583164    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"
Nov 01 01:09:22.583352 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.583252    4493 scope.go:122] "RemoveContainer" containerID="0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"
Nov 01 01:09:22.608402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.608371    4493 scope.go:122] "RemoveContainer" containerID="0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"
Nov 01 01:09:22.608803 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:22.608772    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131\": container with ID starting with 0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131 not found: ID does not exist" containerID="0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"
Nov 01 01:09:22.608871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.608817    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131"} err="failed to get container status \"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131\": rpc error: code = NotFound desc = could not find container \"0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131\": container with ID starting with 0298b826b4285917de95c9e1b619590dc0a0395ae5f20cfb4adacc069b25c131 not found: ID does not exist"
Nov 01 01:09:22.679826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.679797    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fe9570-571e-4a1e-b954-40f64a70a519-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.679826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.679823    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbcn\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-kube-api-access-7nbcn\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.680025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.679836    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fe9570-571e-4a1e-b954-40f64a70a519-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.680025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.679844    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fe9570-571e-4a1e-b954-40f64a70a519-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:22.787977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.787933    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:22.791413 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:22.791387    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-s9nv8"]
Nov 01 01:09:24.506513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:24.506480    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" path="/var/lib/kubelet/pods/20fe9570-571e-4a1e-b954-40f64a70a519/volumes"
Nov 01 01:09:29.715253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.715168    4493 generic.go:356] "Generic (PLEG): container finished" podID="6d393838-b6da-4012-8830-fe231a36bfc9" containerID="7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f" exitCode=0
Nov 01 01:09:29.715677 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.715246    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerDied","Data":"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"}
Nov 01 01:09:29.715778 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.715761    4493 scope.go:122] "RemoveContainer" containerID="7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"
Nov 01 01:09:29.717422 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.717397    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:29.717528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.717464    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:29.717528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:29.717496    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:30.744965 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:30.744930    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerStarted","Data":"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"}
Nov 01 01:09:31.238217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:31.234858    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:09:31.240184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:31.238685    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 01:09:31.240184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:31.238747    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 01:09:31.635216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:31.634248    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:09:32.234160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:32.232511    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 01:09:32.234160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:32.232572    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 01:09:33.236159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:33.231582    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 01:09:33.236159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:33.231646    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 01:09:34.236159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:34.229726    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 01:09:34.236159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:34.229799    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 01:09:35.240557 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:35.240523    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 01:09:35.240992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:35.240579    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 01:09:36.221941 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:36.221910    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:09:36.223109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:36.223092    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:09:44.989621 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:44.989587    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:09:45.081528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:45.081495    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:09:45.189772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:45.189737    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:09:49.057160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057090    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:09:49.057817 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057796    4493 cpu_manager.go:414] "containerMap: removing container" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057818    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057842    4493 cpu_manager.go:414] "containerMap: removing container" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057850    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057884    4493 cpu_manager.go:414] "containerMap: removing container" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057891    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057912    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy"
Nov 01 01:09:49.057919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057918    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057937    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.057947    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.058101    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fe9570-571e-4a1e-b954-40f64a70a519" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.058115    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fc6cd1-3a42-4283-9e71-9540c51e37b6" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.058147    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b92ba3-60ed-4a0e-8067-7daf614d6e7c" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.058159    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8171f6-7f06-4ce7-9d0b-171ab8721445" containerName="istio-proxy"
Nov 01 01:09:49.058316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.058173    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aba9d9f-09d2-4607-9b1c-f8bf4c67408b" containerName="istio-proxy"
Nov 01 01:09:49.062937 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.062917    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087636    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087666    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087767    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087802    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087819    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.087991 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.087895    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.088049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.088009    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhfx\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.088108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.088051    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.088203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.088125    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.106873 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.106846    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:09:49.110620 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.110600    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.134246 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.134225    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\""
Nov 01 01:09:49.172064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.172037    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:09:49.189043 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189005    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189051    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189077    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189105    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhfx\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189125    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189258    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189326    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189378    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189433    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189479    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189544    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189576    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189628    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189698    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189725    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.189806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189754    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2h6k\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.190176 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.189807    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.190176 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.190097    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.190391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.190367    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.190467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.190385    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.190698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.190669    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.192110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.192090    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.192615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.192586    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.247567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.247534    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:09:49.253195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.253111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.253446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.253422    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhfx\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\") pod \"maas-default-gateway-openshift-default-67bf986f99-2xrm7\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.291172 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291114    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2h6k\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291226    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291258    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291403    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291436    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291457    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291485    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291545    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291578    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291710    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.291841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.291817    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.292061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.292020    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.292061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.292035    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.292232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.292213    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.293942 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.293918    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.294073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.294056    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.352561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.352479    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.352714 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.352627    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2h6k\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-t48rm\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.376555 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.376529    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:49.423953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.423923    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:49.809395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.809373    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:09:49.810193 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:09:49.810160    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde838895_1c79_4c6d_bab6_7aca01a66a04.slice/crio-57cc84f0120596362a1822d0a6fd38617c714d521d012fd7a3cb42f00b6fe3d2 WatchSource:0}: Error finding container 57cc84f0120596362a1822d0a6fd38617c714d521d012fd7a3cb42f00b6fe3d2: Status 404 returned error can't find the container with id 57cc84f0120596362a1822d0a6fd38617c714d521d012fd7a3cb42f00b6fe3d2
Nov 01 01:09:49.812757 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.812724    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:49.812863 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.812802    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:49.812863 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.812846    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:49.857807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.857779    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" event={"ID":"de838895-1c79-4c6d-bab6-7aca01a66a04","Type":"ContainerStarted","Data":"57cc84f0120596362a1822d0a6fd38617c714d521d012fd7a3cb42f00b6fe3d2"}
Nov 01 01:09:49.862434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.862409    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:09:49.864110 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:09:49.864089    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adf1d7e_1093_4ab5_9406_88d379726d9f.slice/crio-49808eaa139744295fdd73d88df37d3bfc3d9fca7215b62f71542f5f2ad90d03 WatchSource:0}: Error finding container 49808eaa139744295fdd73d88df37d3bfc3d9fca7215b62f71542f5f2ad90d03: Status 404 returned error can't find the container with id 49808eaa139744295fdd73d88df37d3bfc3d9fca7215b62f71542f5f2ad90d03
Nov 01 01:09:49.866610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.866579    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:49.866710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.866641    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:49.866710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:49.866667    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:09:50.082465 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.082378    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" podUID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" containerName="payload-processing" containerID="cri-o://f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295" gracePeriod=25
Nov 01 01:09:50.190478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.190437    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" podUID="1c51ee02-ca2a-4baf-85f9-eac144474bae" containerName="payload-pre-processing" containerID="cri-o://3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e" gracePeriod=25
Nov 01 01:09:50.407893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.407621    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:09:50.507193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.506925    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\") pod \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") "
Nov 01 01:09:50.507193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.507053    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\" (UniqueName: \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\") pod \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\" (UID: \"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2\") "
Nov 01 01:09:50.507395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.507285    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume" pod "55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" (UID: "55fc0754-fa9d-4e98-b58f-ea562ff6f3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:50.507744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.507517    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:50.509402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.509372    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz" pod "55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" (UID: "55fc0754-fa9d-4e98-b58f-ea562ff6f3c2"). InnerVolumeSpecName "kube-api-access-s7qqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:50.578670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.578577    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:09:50.668217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.638494    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\") pod \"1c51ee02-ca2a-4baf-85f9-eac144474bae\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") "
Nov 01 01:09:50.668217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.638704    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\" (UniqueName: \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\") pod \"1c51ee02-ca2a-4baf-85f9-eac144474bae\" (UID: \"1c51ee02-ca2a-4baf-85f9-eac144474bae\") "
Nov 01 01:09:50.668217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.639208    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7qqz\" (UniqueName: \"kubernetes.io/projected/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2-kube-api-access-s7qqz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:50.668217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.663623    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume" pod "1c51ee02-ca2a-4baf-85f9-eac144474bae" (UID: "1c51ee02-ca2a-4baf-85f9-eac144474bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:09:50.683042 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.683004    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd" pod "1c51ee02-ca2a-4baf-85f9-eac144474bae" (UID: "1c51ee02-ca2a-4baf-85f9-eac144474bae"). InnerVolumeSpecName "kube-api-access-s8gnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:09:50.771882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.771846    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c51ee02-ca2a-4baf-85f9-eac144474bae-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:50.771882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.771873    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gnd\" (UniqueName: \"kubernetes.io/projected/1c51ee02-ca2a-4baf-85f9-eac144474bae-kube-api-access-s8gnd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:09:50.872634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.872593    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" event={"ID":"de838895-1c79-4c6d-bab6-7aca01a66a04","Type":"ContainerStarted","Data":"5bd64dbddac6b655a7e641143bbab9345748cda318337c71ab5b9018b38240d9"}
Nov 01 01:09:50.875640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.875572    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" event={"ID":"4adf1d7e-1093-4ab5-9406-88d379726d9f","Type":"ContainerStarted","Data":"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"}
Nov 01 01:09:50.875801 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.875688    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" event={"ID":"4adf1d7e-1093-4ab5-9406-88d379726d9f","Type":"ContainerStarted","Data":"49808eaa139744295fdd73d88df37d3bfc3d9fca7215b62f71542f5f2ad90d03"}
Nov 01 01:09:50.877791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.877758    4493 generic.go:356] "Generic (PLEG): container finished" podID="1c51ee02-ca2a-4baf-85f9-eac144474bae" containerID="3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e" exitCode=0
Nov 01 01:09:50.877923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.877892    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" event={"ID":"1c51ee02-ca2a-4baf-85f9-eac144474bae","Type":"ContainerDied","Data":"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"}
Nov 01 01:09:50.877980 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.877939    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6" event={"ID":"1c51ee02-ca2a-4baf-85f9-eac144474bae","Type":"ContainerDied","Data":"8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9"}
Nov 01 01:09:50.878032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.877981    4493 scope.go:122] "RemoveContainer" containerID="3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"
Nov 01 01:09:50.878229 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.878212    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"
Nov 01 01:09:50.880738 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.880716    4493 generic.go:356] "Generic (PLEG): container finished" podID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" containerID="f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295" exitCode=0
Nov 01 01:09:50.880842 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.880799    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" event={"ID":"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2","Type":"ContainerDied","Data":"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"}
Nov 01 01:09:50.880944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.880835    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f" event={"ID":"55fc0754-fa9d-4e98-b58f-ea562ff6f3c2","Type":"ContainerDied","Data":"721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4"}
Nov 01 01:09:50.881170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.881156    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"
Nov 01 01:09:50.900575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.900554    4493 scope.go:122] "RemoveContainer" containerID="3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"
Nov 01 01:09:50.901355 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:50.901334    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e\": container with ID starting with 3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e not found: ID does not exist" containerID="3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"
Nov 01 01:09:50.901498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.901479    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e"} err="failed to get container status \"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e\": rpc error: code = NotFound desc = could not find container \"3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e\": container with ID starting with 3b60f991e9b31809b431e826aa9aee02d3f2d13563aa94e792ba49eb7be3414e not found: ID does not exist"
Nov 01 01:09:50.901575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.901566    4493 scope.go:122] "RemoveContainer" containerID="f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"
Nov 01 01:09:50.910446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.910432    4493 scope.go:122] "RemoveContainer" containerID="f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"
Nov 01 01:09:50.910731 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:50.910714    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295\": container with ID starting with f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295 not found: ID does not exist" containerID="f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"
Nov 01 01:09:50.910821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.910804    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295"} err="failed to get container status \"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295\": rpc error: code = NotFound desc = could not find container \"f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295\": container with ID starting with f300ae3bf5be18ba7f1fce2a6e10ba3ca57faf47cb0b060c130e2d150d5d3295 not found: ID does not exist"
Nov 01 01:09:50.983161 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:09:50.983108    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c51ee02_ca2a_4baf_85f9_eac144474bae.slice/crio-8c0838deb9a4b03a579eed6ab2eac4b64df3cf05f312e4333e9f428c8a1a54b9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c51ee02_ca2a_4baf_85f9_eac144474bae.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fc0754_fa9d_4e98_b58f_ea562ff6f3c2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fc0754_fa9d_4e98_b58f_ea562ff6f3c2.slice/crio-721ee4ededc457a5b5a9babbd23b76951c3e7f8ddd98bf04ff8ab55cb0691ae4\": RecentStats: unable to find data in memory cache]"
Nov 01 01:09:50.997574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:50.997527    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podStartSLOduration=1.9975136039999999 podStartE2EDuration="1.997513604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:50.995578542 +0000 UTC m=+2780.673165487" watchObservedRunningTime="2026-11-01 01:09:50.997513604 +0000 UTC m=+2780.675100549"
Nov 01 01:09:51.144247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.144200    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:09:51.146458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.146419    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-718e19-59bbbc8fd8-6nz9f"]
Nov 01 01:09:51.248221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.244912    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podStartSLOduration=2.244889741 podStartE2EDuration="2.244889741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:51.241685697 +0000 UTC m=+2780.919272644" watchObservedRunningTime="2026-11-01 01:09:51.244889741 +0000 UTC m=+2780.922476688"
Nov 01 01:09:51.361240 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.361195    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:09:51.366678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.366644    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-718e19-7f47fbf4f5-pdvb6"]
Nov 01 01:09:51.377463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.377435    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:51.379462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.379436    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:51.379578 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.379498    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:51.431123 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.431092    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:51.431567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.431538    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:51.431732 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:51.431711    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:52.377023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.376987    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:52.377462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.377038    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:52.430161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.428417    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:52.430161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.428478    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:52.503020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.502985    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c51ee02-ca2a-4baf-85f9-eac144474bae" path="/var/lib/kubelet/pods/1c51ee02-ca2a-4baf-85f9-eac144474bae/volumes"
Nov 01 01:09:52.503480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:52.503463    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" path="/var/lib/kubelet/pods/55fc0754-fa9d-4e98-b58f-ea562ff6f3c2/volumes"
Nov 01 01:09:53.377375 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:53.377339    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:53.377754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:53.377403    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:53.424563 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:53.424524    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:53.424845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:53.424817    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:54.377714 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:54.377684    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:54.378228 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:54.378203    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:54.427160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:54.424432    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:54.427160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:54.424494    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:55.377493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:55.377451    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:55.377708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:55.377529    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:55.424609 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:55.424571    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:55.425239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:55.425206    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:56.377507 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:56.377471    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:56.377693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:56.377545    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:56.425374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:56.425338    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:56.425753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:56.425406    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:57.377482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:57.377442    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2xrm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 01 01:09:57.377715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:57.377522    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 01 01:09:57.425257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:57.425224    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 01:09:57.425435 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:57.425279    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 01:09:58.426126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.426084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:58.427428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.427397    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:58.429695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.429667    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:09:58.561966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.561924    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:58.562496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.562473    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:09:58.563424 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:09:58.563409    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:10:04.228126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228094    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:10:04.228735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228719    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1c51ee02-ca2a-4baf-85f9-eac144474bae" containerName="payload-pre-processing"
Nov 01 01:10:04.228812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228736    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c51ee02-ca2a-4baf-85f9-eac144474bae" containerName="payload-pre-processing"
Nov 01 01:10:04.228812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228759    4493 cpu_manager.go:414] "containerMap: removing container" podUID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" containerName="payload-processing"
Nov 01 01:10:04.228812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228768    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" containerName="payload-processing"
Nov 01 01:10:04.228962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228888    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c51ee02-ca2a-4baf-85f9-eac144474bae" containerName="payload-pre-processing"
Nov 01 01:10:04.228962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.228906    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fc0754-fa9d-4e98-b58f-ea562ff6f3c2" containerName="payload-processing"
Nov 01 01:10:04.232499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.232479    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.276600 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.276562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.276930 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.276906    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.276975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277089 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277067    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277157    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277228    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjxj\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277326    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.277435 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.277420    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.339361 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.339323    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:10:04.378432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378397    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378437    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378484    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378569    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378622    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378686    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378715    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378761    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.378861 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378808    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjxj\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.379072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.378936    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.379262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.379236    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.379408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.379239    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.379408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.379305    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.379664 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.379637    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.381287 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.381264    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.381583 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.381563    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.440650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.440619    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.440833 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.440761    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjxj\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\") pod \"maas-default-gateway-openshift-default-67bf986f99-p9ldb\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.549389 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.549313    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:04.974516 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.974486    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:10:04.987183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:04.987149    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:10:05.024574 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:10:05.024544    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b77912c_ade6_4469_a0d7_4b5c05e72226.slice/crio-95ee21442a1a66902207b5bb99bae3793623dcf95a4350b087de93631a46fb4c WatchSource:0}: Error finding container 95ee21442a1a66902207b5bb99bae3793623dcf95a4350b087de93631a46fb4c: Status 404 returned error can't find the container with id 95ee21442a1a66902207b5bb99bae3793623dcf95a4350b087de93631a46fb4c
Nov 01 01:10:05.027326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:05.027284    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:05.027404 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:05.027391    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:05.027447 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:05.027437    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:05.977329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:05.976549    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" event={"ID":"8b77912c-ade6-4469-a0d7-4b5c05e72226","Type":"ContainerStarted","Data":"084e96efa151adfa70f1b43bac55e5474f40425ef567af0f21124f38d9b82117"}
Nov 01 01:10:05.977329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:05.976605    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" event={"ID":"8b77912c-ade6-4469-a0d7-4b5c05e72226","Type":"ContainerStarted","Data":"95ee21442a1a66902207b5bb99bae3793623dcf95a4350b087de93631a46fb4c"}
Nov 01 01:10:06.207430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:06.205015    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podStartSLOduration=2.2050008500000002 podStartE2EDuration="2.20500085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:10:06.153613789 +0000 UTC m=+2795.831200736" watchObservedRunningTime="2026-11-01 01:10:06.20500085 +0000 UTC m=+2795.882587866"
Nov 01 01:10:06.552158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:06.551361    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:06.571347 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:06.571259    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:06.571347 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:06.571324    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:07.550560 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:07.550513    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:07.551034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:07.550584    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:08.551160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:08.550429    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:08.551160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:08.550491    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:09.556207 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:09.555587    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:09.556207 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:09.555652    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:10.563158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:10.562531    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:10.563158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:10.562593    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:11.549993 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:11.549953    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:11.550190 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:11.550021    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:12.550666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:12.550623    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p9ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 01:10:12.551149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:12.550700    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 01:10:13.822260 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:13.822225    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:13.822773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:13.822582    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:13.823827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:13.823807    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:10:14.989625 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:14.989583    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:10:15.095834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.095803    4493 generic.go:356] "Generic (PLEG): container finished" podID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerID="21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db" exitCode=0
Nov 01 01:10:15.096017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.095877    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerDied","Data":"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"}
Nov 01 01:10:15.096434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.096408    4493 scope.go:122] "RemoveContainer" containerID="21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"
Nov 01 01:10:15.097672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.097646    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:15.097764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.097711    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:15.097764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.097742    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:10:15.142014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.141992    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:10:15.142208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.142188    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" podUID="00dc6468-96e7-47bd-8e46-2422ff75e562" containerName="maas-api" containerID="cri-o://c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e" gracePeriod=30
Nov 01 01:10:15.436328 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.436304    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:10:15.633796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.622059    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\" (UniqueName: \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\") pod \"00dc6468-96e7-47bd-8e46-2422ff75e562\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") "
Nov 01 01:10:15.633796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.622237    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\") pod \"00dc6468-96e7-47bd-8e46-2422ff75e562\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") "
Nov 01 01:10:15.633796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.622298    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") pod \"00dc6468-96e7-47bd-8e46-2422ff75e562\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") "
Nov 01 01:10:15.633796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.622347    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\") pod \"00dc6468-96e7-47bd-8e46-2422ff75e562\" (UID: \"00dc6468-96e7-47bd-8e46-2422ff75e562\") "
Nov 01 01:10:15.640448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.640402    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca" pod "00dc6468-96e7-47bd-8e46-2422ff75e562" (UID: "00dc6468-96e7-47bd-8e46-2422ff75e562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:15.641002 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.640971    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs" pod "00dc6468-96e7-47bd-8e46-2422ff75e562" (UID: "00dc6468-96e7-47bd-8e46-2422ff75e562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:15.641752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.641723    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls" pod "00dc6468-96e7-47bd-8e46-2422ff75e562" (UID: "00dc6468-96e7-47bd-8e46-2422ff75e562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:15.643022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.643000    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr" pod "00dc6468-96e7-47bd-8e46-2422ff75e562" (UID: "00dc6468-96e7-47bd-8e46-2422ff75e562"). InnerVolumeSpecName "kube-api-access-xqwgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:15.740304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.740269    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:15.740304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.740308    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dc6468-96e7-47bd-8e46-2422ff75e562-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:15.740514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.740326    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dc6468-96e7-47bd-8e46-2422ff75e562-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:15.740514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:15.740341    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwgr\" (UniqueName: \"kubernetes.io/projected/00dc6468-96e7-47bd-8e46-2422ff75e562-kube-api-access-xqwgr\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:16.104421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.104388    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerStarted","Data":"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"}
Nov 01 01:10:16.105696 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.105676    4493 generic.go:356] "Generic (PLEG): container finished" podID="00dc6468-96e7-47bd-8e46-2422ff75e562" containerID="c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e" exitCode=0
Nov 01 01:10:16.105792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.105711    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" event={"ID":"00dc6468-96e7-47bd-8e46-2422ff75e562","Type":"ContainerDied","Data":"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"}
Nov 01 01:10:16.105792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.105714    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"
Nov 01 01:10:16.105792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.105746    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5" event={"ID":"00dc6468-96e7-47bd-8e46-2422ff75e562","Type":"ContainerDied","Data":"40d3ccaf1ac633c07fb7f5eea59db8b7639278a0a9f52a0ac6eb84cb4b446bdd"}
Nov 01 01:10:16.105792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.105765    4493 scope.go:122] "RemoveContainer" containerID="c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"
Nov 01 01:10:16.138372 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.138340    4493 scope.go:122] "RemoveContainer" containerID="c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"
Nov 01 01:10:16.141732 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:16.141689    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e\": container with ID starting with c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e not found: ID does not exist" containerID="c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"
Nov 01 01:10:16.141897 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.141741    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e"} err="failed to get container status \"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e\": rpc error: code = NotFound desc = could not find container \"c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e\": container with ID starting with c327599ada906e1a7596fde5bd7a2504387c03aea103ec48d2193cea35e23c6e not found: ID does not exist"
Nov 01 01:10:16.366253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.362259    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:10:16.366704 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.366623    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-603e8c-7675b76bfd-vnmp5"]
Nov 01 01:10:16.503573 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:16.503531    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dc6468-96e7-47bd-8e46-2422ff75e562" path="/var/lib/kubelet/pods/00dc6468-96e7-47bd-8e46-2422ff75e562/volumes"
Nov 01 01:10:17.005507 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:17.005472    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:10:17.005718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:17.005618    4493 patch_prober.go:29] interesting pod/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 01:10:17.005718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:17.005682    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 01:10:18.006047 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:18.006014    4493 patch_prober.go:29] interesting pod/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 01:10:18.006510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:18.006079    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 01:10:19.005907 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:19.005871    4493 patch_prober.go:29] interesting pod/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 01:10:19.006083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:19.005940    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 01:10:20.005783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:20.005747    4493 patch_prober.go:29] interesting pod/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 01:10:20.005969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:20.005804    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 01:10:21.006327 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:21.006295    4493 patch_prober.go:29] interesting pod/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 01:10:21.006816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:21.006347    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 01:10:22.011005 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:22.010960    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:10:22.011587 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:22.011361    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:10:22.012101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:22.012084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:10:29.647836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:29.647804    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:10:29.764284 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:29.764252    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:10:34.648785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:34.648734    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" podUID="a9bfcc4a-8362-45cc-b943-b450b214d1df" containerName="payload-processing" containerID="cri-o://c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8" gracePeriod=25
Nov 01 01:10:34.765499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:34.765463    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" podUID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" containerName="payload-pre-processing" containerID="cri-o://8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a" gracePeriod=25
Nov 01 01:10:34.918668 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:34.918596    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:10:35.048637 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.048608    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:10:35.112774 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.112735    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\") pod \"a9bfcc4a-8362-45cc-b943-b450b214d1df\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") "
Nov 01 01:10:35.112976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.112863    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\" (UniqueName: \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\") pod \"a9bfcc4a-8362-45cc-b943-b450b214d1df\" (UID: \"a9bfcc4a-8362-45cc-b943-b450b214d1df\") "
Nov 01 01:10:35.113289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.113250    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume" pod "a9bfcc4a-8362-45cc-b943-b450b214d1df" (UID: "a9bfcc4a-8362-45cc-b943-b450b214d1df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:35.113409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.113383    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bfcc4a-8362-45cc-b943-b450b214d1df-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:35.115261 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.115237    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m" pod "a9bfcc4a-8362-45cc-b943-b450b214d1df" (UID: "a9bfcc4a-8362-45cc-b943-b450b214d1df"). InnerVolumeSpecName "kube-api-access-t659m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:35.213992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.213915    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\" (UniqueName: \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\") pod \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") "
Nov 01 01:10:35.214208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.214021    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\") pod \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\" (UID: \"ba16e7ce-afd5-4491-97d7-40bd174e4f28\") "
Nov 01 01:10:35.214382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.214357    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t659m\" (UniqueName: \"kubernetes.io/projected/a9bfcc4a-8362-45cc-b943-b450b214d1df-kube-api-access-t659m\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:35.214446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.214375    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume" pod "ba16e7ce-afd5-4491-97d7-40bd174e4f28" (UID: "ba16e7ce-afd5-4491-97d7-40bd174e4f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:35.216245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.216222    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh" pod "ba16e7ce-afd5-4491-97d7-40bd174e4f28" (UID: "ba16e7ce-afd5-4491-97d7-40bd174e4f28"). InnerVolumeSpecName "kube-api-access-k9clh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:35.272855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.272818    4493 generic.go:356] "Generic (PLEG): container finished" podID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" containerID="8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a" exitCode=0
Nov 01 01:10:35.273017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.272885    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"
Nov 01 01:10:35.273017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.272907    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" event={"ID":"ba16e7ce-afd5-4491-97d7-40bd174e4f28","Type":"ContainerDied","Data":"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"}
Nov 01 01:10:35.273017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.273004    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc" event={"ID":"ba16e7ce-afd5-4491-97d7-40bd174e4f28","Type":"ContainerDied","Data":"abb67cc484f5b1ae761ed3a87253a239b32eb4d4898fffef0f808b89c52983ad"}
Nov 01 01:10:35.273203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.273028    4493 scope.go:122] "RemoveContainer" containerID="8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"
Nov 01 01:10:35.274453 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.274436    4493 generic.go:356] "Generic (PLEG): container finished" podID="a9bfcc4a-8362-45cc-b943-b450b214d1df" containerID="c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8" exitCode=0
Nov 01 01:10:35.274582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.274565    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" event={"ID":"a9bfcc4a-8362-45cc-b943-b450b214d1df","Type":"ContainerDied","Data":"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"}
Nov 01 01:10:35.274624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.274605    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm" event={"ID":"a9bfcc4a-8362-45cc-b943-b450b214d1df","Type":"ContainerDied","Data":"f89d72cf49ab6d76a25b54f6ee69f9c640a3f7339f675119762f5b76005f2416"}
Nov 01 01:10:35.274688 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.274677    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"
Nov 01 01:10:35.284772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.284751    4493 scope.go:122] "RemoveContainer" containerID="8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"
Nov 01 01:10:35.285116 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:35.285074    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a\": container with ID starting with 8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a not found: ID does not exist" containerID="8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"
Nov 01 01:10:35.285226 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.285119    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a"} err="failed to get container status \"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a\": rpc error: code = NotFound desc = could not find container \"8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a\": container with ID starting with 8714f28a74e8d65d3f50d3775aba9a2e136a20686c8cc6b1cabff551a761e84a not found: ID does not exist"
Nov 01 01:10:35.285226 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.285159    4493 scope.go:122] "RemoveContainer" containerID="c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"
Nov 01 01:10:35.293821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.293801    4493 scope.go:122] "RemoveContainer" containerID="c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"
Nov 01 01:10:35.294059 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:35.294041    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8\": container with ID starting with c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8 not found: ID does not exist" containerID="c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"
Nov 01 01:10:35.294125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.294070    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8"} err="failed to get container status \"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8\": rpc error: code = NotFound desc = could not find container \"c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8\": container with ID starting with c76a47e55167e9a58f61dfd7d04310fe9835b07b53c337fbd29808d8f6d716f8 not found: ID does not exist"
Nov 01 01:10:35.314992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.314967    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9clh\" (UniqueName: \"kubernetes.io/projected/ba16e7ce-afd5-4491-97d7-40bd174e4f28-kube-api-access-k9clh\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:35.314992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.314991    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba16e7ce-afd5-4491-97d7-40bd174e4f28-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:35.445676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.445645    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:10:35.451810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.451784    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9b61-795b4fc4c8-zmkbm"]
Nov 01 01:10:35.563346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.563301    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:10:35.567486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:35.567459    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9b61-55466c849f-pwxqc"]
Nov 01 01:10:36.502348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:36.502316    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bfcc4a-8362-45cc-b943-b450b214d1df" path="/var/lib/kubelet/pods/a9bfcc4a-8362-45cc-b943-b450b214d1df/volumes"
Nov 01 01:10:36.502735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:36.502722    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" path="/var/lib/kubelet/pods/ba16e7ce-afd5-4491-97d7-40bd174e4f28/volumes"
Nov 01 01:10:36.765970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:36.765892    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 01:10:36.780506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:36.780469    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 01:10:37.741596 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:37.741563    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:10:37.859462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:37.859433    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:10:40.352558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:40.352524    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:10:42.742451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:42.742206    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" podUID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" containerName="payload-processing" containerID="cri-o://feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65" gracePeriod=25
Nov 01 01:10:42.866168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:42.863203    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerName="payload-pre-processing" containerID="cri-o://09aec913edc20c7f0d474fbf202285b13926614f5f476ad6b9379b4ab8cccf4a" gracePeriod=25
Nov 01 01:10:43.240158 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:10:43.234417    4493 logging.go:55] [core] [Channel #5789 SubChannel #5790]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.143:50052", ServerName: "10.133.0.143:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.143:50052: connect: connection refused"
Nov 01 01:10:43.503669 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.503639    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:10:43.508004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.507978    4493 generic.go:356] "Generic (PLEG): container finished" podID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerID="09aec913edc20c7f0d474fbf202285b13926614f5f476ad6b9379b4ab8cccf4a" exitCode=0
Nov 01 01:10:43.508151 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.508049    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" event={"ID":"dd55d2f0-a035-46dd-989f-cbc1063bc253","Type":"ContainerDied","Data":"09aec913edc20c7f0d474fbf202285b13926614f5f476ad6b9379b4ab8cccf4a"}
Nov 01 01:10:43.509788 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.509763    4493 generic.go:356] "Generic (PLEG): container finished" podID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" containerID="feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65" exitCode=0
Nov 01 01:10:43.509905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.509813    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"
Nov 01 01:10:43.509905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.509870    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" event={"ID":"bc099aae-5b39-494e-8a68-ec4b8ff3743e","Type":"ContainerDied","Data":"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"}
Nov 01 01:10:43.510016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.509907    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg" event={"ID":"bc099aae-5b39-494e-8a68-ec4b8ff3743e","Type":"ContainerDied","Data":"753aff6c58cc6ac585f21d9637c5ed9f669a774e62fda7726eaae846e818bebd"}
Nov 01 01:10:43.510016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.509959    4493 scope.go:122] "RemoveContainer" containerID="feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"
Nov 01 01:10:43.529758 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.515125    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\") pod \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") "
Nov 01 01:10:43.530299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.530262    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\" (UniqueName: \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\") pod \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\" (UID: \"bc099aae-5b39-494e-8a68-ec4b8ff3743e\") "
Nov 01 01:10:43.550893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.550812    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume" pod "bc099aae-5b39-494e-8a68-ec4b8ff3743e" (UID: "bc099aae-5b39-494e-8a68-ec4b8ff3743e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:43.556227 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.551227    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf" pod "bc099aae-5b39-494e-8a68-ec4b8ff3743e" (UID: "bc099aae-5b39-494e-8a68-ec4b8ff3743e"). InnerVolumeSpecName "kube-api-access-76wgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:43.593811 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.593785    4493 scope.go:122] "RemoveContainer" containerID="feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"
Nov 01 01:10:43.594630 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:43.594588    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65\": container with ID starting with feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65 not found: ID does not exist" containerID="feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"
Nov 01 01:10:43.594630 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.594627    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65"} err="failed to get container status \"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65\": rpc error: code = NotFound desc = could not find container \"feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65\": container with ID starting with feb8aadcaaeff496c5ef6066d614c7f51474a792e1ff7845d78777b3feb10d65 not found: ID does not exist"
Nov 01 01:10:43.603539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.603520    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:10:43.638737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.638707    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\" (UniqueName: \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\") pod \"dd55d2f0-a035-46dd-989f-cbc1063bc253\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") "
Nov 01 01:10:43.639484 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.639460    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\") pod \"dd55d2f0-a035-46dd-989f-cbc1063bc253\" (UID: \"dd55d2f0-a035-46dd-989f-cbc1063bc253\") "
Nov 01 01:10:43.640217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.640194    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wgf\" (UniqueName: \"kubernetes.io/projected/bc099aae-5b39-494e-8a68-ec4b8ff3743e-kube-api-access-76wgf\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:43.640356 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.640340    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc099aae-5b39-494e-8a68-ec4b8ff3743e-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:43.652786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.652756    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v" pod "dd55d2f0-a035-46dd-989f-cbc1063bc253" (UID: "dd55d2f0-a035-46dd-989f-cbc1063bc253"). InnerVolumeSpecName "kube-api-access-9f57v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:43.658391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.658354    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume" pod "dd55d2f0-a035-46dd-989f-cbc1063bc253" (UID: "dd55d2f0-a035-46dd-989f-cbc1063bc253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:43.776418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.776340    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f57v\" (UniqueName: \"kubernetes.io/projected/dd55d2f0-a035-46dd-989f-cbc1063bc253-kube-api-access-9f57v\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:43.776418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.776382    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd55d2f0-a035-46dd-989f-cbc1063bc253-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:43.795153 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.788520    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:10:43.796436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:43.795777    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" podUID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" containerName="maas-api" containerID="cri-o://f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97" gracePeriod=30
Nov 01 01:10:44.031952 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.031920    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 01:10:44.086303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.086225    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:10:44.091219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.091196    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-030da8-6db6f68668-wszfg"]
Nov 01 01:10:44.115690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.115670    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:10:44.182723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.182692    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\") pod \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") "
Nov 01 01:10:44.182868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.182750    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\") pod \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") "
Nov 01 01:10:44.182868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.182833    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") pod \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") "
Nov 01 01:10:44.182958 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.182911    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\" (UniqueName: \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\") pod \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\" (UID: \"ff57b037-ade1-450a-a893-7d5f1c8bbe0c\") "
Nov 01 01:10:44.183169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.183126    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca" pod "ff57b037-ade1-450a-a893-7d5f1c8bbe0c" (UID: "ff57b037-ade1-450a-a893-7d5f1c8bbe0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:44.183374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.183355    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:44.185155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.185105    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs" pod "ff57b037-ade1-450a-a893-7d5f1c8bbe0c" (UID: "ff57b037-ade1-450a-a893-7d5f1c8bbe0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:44.185155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.185116    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq" pod "ff57b037-ade1-450a-a893-7d5f1c8bbe0c" (UID: "ff57b037-ade1-450a-a893-7d5f1c8bbe0c"). InnerVolumeSpecName "kube-api-access-wcmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:44.185155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.185123    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls" pod "ff57b037-ade1-450a-a893-7d5f1c8bbe0c" (UID: "ff57b037-ade1-450a-a893-7d5f1c8bbe0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:44.224054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.224020    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.143:50052\" within 1s: context deadline exceeded"
Nov 01 01:10:44.286158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.286056    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:44.286158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.286105    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:44.286158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.286145    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcmcq\" (UniqueName: \"kubernetes.io/projected/ff57b037-ade1-450a-a893-7d5f1c8bbe0c-kube-api-access-wcmcq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:44.502801 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.502771    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" path="/var/lib/kubelet/pods/bc099aae-5b39-494e-8a68-ec4b8ff3743e/volumes"
Nov 01 01:10:44.516588 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.516566    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"
Nov 01 01:10:44.516726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.516598    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw" event={"ID":"dd55d2f0-a035-46dd-989f-cbc1063bc253","Type":"ContainerDied","Data":"9e2a4353c7807898f9cdc863c4475089837def292168b7e0b7d3705665cf375d"}
Nov 01 01:10:44.516726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.516653    4493 scope.go:122] "RemoveContainer" containerID="09aec913edc20c7f0d474fbf202285b13926614f5f476ad6b9379b4ab8cccf4a"
Nov 01 01:10:44.517988 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.517961    4493 generic.go:356] "Generic (PLEG): container finished" podID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" containerID="f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97" exitCode=0
Nov 01 01:10:44.518085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.518004    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"
Nov 01 01:10:44.518085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.518042    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" event={"ID":"ff57b037-ade1-450a-a893-7d5f1c8bbe0c","Type":"ContainerDied","Data":"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"}
Nov 01 01:10:44.518189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.518085    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57" event={"ID":"ff57b037-ade1-450a-a893-7d5f1c8bbe0c","Type":"ContainerDied","Data":"5638e794785645bf0c9eab8db3b04a4844188bc0fd973f102767b71a86b01b31"}
Nov 01 01:10:44.526208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.526187    4493 scope.go:122] "RemoveContainer" containerID="f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"
Nov 01 01:10:44.535310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.535284    4493 scope.go:122] "RemoveContainer" containerID="f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"
Nov 01 01:10:44.535592 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:44.535571    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97\": container with ID starting with f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97 not found: ID does not exist" containerID="f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"
Nov 01 01:10:44.535646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.535602    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97"} err="failed to get container status \"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97\": rpc error: code = NotFound desc = could not find container \"f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97\": container with ID starting with f7aaccd0c3f82ae6f2086eea713f7711a74f7b2eb695124cf445d944f092df97 not found: ID does not exist"
Nov 01 01:10:44.716158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.715191    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:10:44.724160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.723889    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-030da8-855498c747-7j9nw"]
Nov 01 01:10:44.817517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.817429    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:10:44.821681 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:44.821656    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-718e19-74d6749b96-jpm57"]
Nov 01 01:10:44.989599 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:44.989564    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:10:46.502696 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:46.502659    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" path="/var/lib/kubelet/pods/dd55d2f0-a035-46dd-989f-cbc1063bc253/volumes"
Nov 01 01:10:46.503312 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:46.503292    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" path="/var/lib/kubelet/pods/ff57b037-ade1-450a-a893-7d5f1c8bbe0c/volumes"
Nov 01 01:10:49.032242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:49.032178    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing" containerID="cri-o://9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd" gracePeriod=25
Nov 01 01:10:50.323622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:50.323580    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:10:51.786955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:51.786915    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" containerID="cri-o://f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba" gracePeriod=45
Nov 01 01:10:51.800481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:51.800448    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main" containerID="cri-o://9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e" gracePeriod=45
Nov 01 01:10:51.939531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:51.939477    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused"
Nov 01 01:10:52.064966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.064941    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 01:10:52.095408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.095382    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 01:10:52.212655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212625    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.212854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212708    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.212854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212762    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.212854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212801    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.212854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212834    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.213095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212894    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\" (UniqueName: \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.213095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212944    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.213095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.212976    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.213095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.213003    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.213095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.213051    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.213410 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.213111    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\") pod \"b4c8c600-6f30-4c6b-bf55-281ae7918492\" (UID: \"b4c8c600-6f30-4c6b-bf55-281ae7918492\") "
Nov 01 01:10:52.213410 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.213203    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\" (UniqueName: \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\") pod \"2d132881-21f5-425f-b2c3-81af132525d2\" (UID: \"2d132881-21f5-425f-b2c3-81af132525d2\") "
Nov 01 01:10:52.214646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.214350    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.214646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.214394    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.214646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.214600    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.214873 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.214819    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.215499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.215446    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.216240 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.215794    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.216423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.216397    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "kube-api-access-6hj6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:52.216607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.216586    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:52.226321 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.226293    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.227220 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.226623    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:52.227345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.226589    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hj6v\" (UniqueName: \"kubernetes.io/projected/b4c8c600-6f30-4c6b-bf55-281ae7918492-kube-api-access-6hj6v\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227365    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227398    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227414    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227430    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d132881-21f5-425f-b2c3-81af132525d2-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227444    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227464    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.227593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.227478    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.230334 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.230306    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs" pod "b4c8c600-6f30-4c6b-bf55-281ae7918492" (UID: "b4c8c600-6f30-4c6b-bf55-281ae7918492"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:10:52.230698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.230626    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv" pod "2d132881-21f5-425f-b2c3-81af132525d2" (UID: "2d132881-21f5-425f-b2c3-81af132525d2"). InnerVolumeSpecName "kube-api-access-5czqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:52.328685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.328611    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d132881-21f5-425f-b2c3-81af132525d2-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.328685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.328640    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c8c600-6f30-4c6b-bf55-281ae7918492-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.328685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.328651    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czqv\" (UniqueName: \"kubernetes.io/projected/2d132881-21f5-425f-b2c3-81af132525d2-kube-api-access-5czqv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.328685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.328662    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c8c600-6f30-4c6b-bf55-281ae7918492-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:52.593205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.593092    4493 generic.go:356] "Generic (PLEG): container finished" podID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerID="9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e" exitCode=0
Nov 01 01:10:52.593205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.593178    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"
Nov 01 01:10:52.593205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.593182    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" event={"ID":"b4c8c600-6f30-4c6b-bf55-281ae7918492","Type":"ContainerDied","Data":"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"}
Nov 01 01:10:52.593469 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.593230    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb" event={"ID":"b4c8c600-6f30-4c6b-bf55-281ae7918492","Type":"ContainerDied","Data":"c357a5206a359e6b2b287c609a12a649fa5f7a6aaba29ef936ddf2fd1c021c39"}
Nov 01 01:10:52.593469 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.593250    4493 scope.go:122] "RemoveContainer" containerID="9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"
Nov 01 01:10:52.594591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.594572    4493 generic.go:356] "Generic (PLEG): container finished" podID="2d132881-21f5-425f-b2c3-81af132525d2" containerID="f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba" exitCode=0
Nov 01 01:10:52.594688 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.594635    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"
Nov 01 01:10:52.594688 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.594649    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" event={"ID":"2d132881-21f5-425f-b2c3-81af132525d2","Type":"ContainerDied","Data":"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"}
Nov 01 01:10:52.594785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.594686    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6" event={"ID":"2d132881-21f5-425f-b2c3-81af132525d2","Type":"ContainerDied","Data":"d3826d807f65330a5cbe21a9cd49db5689d385be455a1b41b7208f29965d4354"}
Nov 01 01:10:52.602698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.602682    4493 scope.go:122] "RemoveContainer" containerID="9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"
Nov 01 01:10:52.602988 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:52.602970    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e\": container with ID starting with 9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e not found: ID does not exist" containerID="9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"
Nov 01 01:10:52.603035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.602996    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e"} err="failed to get container status \"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e\": rpc error: code = NotFound desc = could not find container \"9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e\": container with ID starting with 9ba4a7c5cb2e4462e501200fd91390d511b9bbb489a12e122fcc8f6f2809dd2e not found: ID does not exist"
Nov 01 01:10:52.603035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.603015    4493 scope.go:122] "RemoveContainer" containerID="f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"
Nov 01 01:10:52.613403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.613387    4493 scope.go:122] "RemoveContainer" containerID="f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"
Nov 01 01:10:52.613678 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:52.613662    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba\": container with ID starting with f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba not found: ID does not exist" containerID="f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"
Nov 01 01:10:52.613724 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.613686    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba"} err="failed to get container status \"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba\": rpc error: code = NotFound desc = could not find container \"f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba\": container with ID starting with f917e59890a826b24d888af5a0d97e7d179fa76a78f2fde5afbd0013bebbbeba not found: ID does not exist"
Nov 01 01:10:52.773029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.773002    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 01:10:52.822613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.822582    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-w3-5a9c19-kserve-7db9467bd6-4jpp6"]
Nov 01 01:10:52.885431 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.885342    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 01:10:52.893334 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:52.893305    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/premium-w3-5a9c19-kserve-6cd969db4c-4gwfb"]
Nov 01 01:10:53.980508 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:53.980474    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:10:53.980913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:53.980698    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" containerID="cri-o://e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71" gracePeriod=30
Nov 01 01:10:54.503273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:54.503241    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d132881-21f5-425f-b2c3-81af132525d2" path="/var/lib/kubelet/pods/2d132881-21f5-425f-b2c3-81af132525d2/volumes"
Nov 01 01:10:54.503704 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:54.503690    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" path="/var/lib/kubelet/pods/b4c8c600-6f30-4c6b-bf55-281ae7918492/volumes"
Nov 01 01:10:59.242530 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.242471    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:10:59.320653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320616    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.320845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320665    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.320845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320714    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.320845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320756    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.320845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320814    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.321072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320869    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.321072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.321072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.320973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.321072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321060    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\") pod \"d055b277-3003-4362-8457-2852717632b3\" (UID: \"d055b277-3003-4362-8457-2852717632b3\") "
Nov 01 01:10:59.321330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321063    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:59.321330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:59.321330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321115    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:59.321330 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321232    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:59.321538 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321477    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:10:59.321651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321631    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.321698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321658    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055b277-3003-4362-8457-2852717632b3-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.321698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321674    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.321698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321687    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.321832 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.321701    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.323168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.323125    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:59.323409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.323388    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:10:59.323722 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.323695    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "kube-api-access-7zgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:10:59.323801 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.323698    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo" pod "d055b277-3003-4362-8457-2852717632b3" (UID: "d055b277-3003-4362-8457-2852717632b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:10:59.422801 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.422767    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055b277-3003-4362-8457-2852717632b3-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.422801 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.422800    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055b277-3003-4362-8457-2852717632b3-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.423040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.422812    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zgzp\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-kube-api-access-7zgzp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.423040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.422826    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055b277-3003-4362-8457-2852717632b3-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:10:59.635042 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.634961    4493 generic.go:356] "Generic (PLEG): container finished" podID="d055b277-3003-4362-8457-2852717632b3" containerID="e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71" exitCode=0
Nov 01 01:10:59.635042 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.635008    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"
Nov 01 01:10:59.635263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.635040    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerDied","Data":"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"}
Nov 01 01:10:59.635263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.635104    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" event={"ID":"d055b277-3003-4362-8457-2852717632b3","Type":"ContainerDied","Data":"417d5aba4ff1688c2efc96924ef1f27b6955537aa365a0ab6c7a78df549bfbf2"}
Nov 01 01:10:59.635263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.635152    4493 scope.go:122] "RemoveContainer" containerID="e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"
Nov 01 01:10:59.645339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.645321    4493 scope.go:122] "RemoveContainer" containerID="4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"
Nov 01 01:10:59.653953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.653928    4493 scope.go:122] "RemoveContainer" containerID="e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"
Nov 01 01:10:59.654245 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:59.654225    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71\": container with ID starting with e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71 not found: ID does not exist" containerID="e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"
Nov 01 01:10:59.654303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.654255    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71"} err="failed to get container status \"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71\": rpc error: code = NotFound desc = could not find container \"e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71\": container with ID starting with e9f8bfc8009937ac70345f68c427b641bb8fc0a00d3d20aabc997b56ad878d71 not found: ID does not exist"
Nov 01 01:10:59.654303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.654273    4493 scope.go:122] "RemoveContainer" containerID="4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"
Nov 01 01:10:59.654486 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:10:59.654468    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00\": container with ID starting with 4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00 not found: ID does not exist" containerID="4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"
Nov 01 01:10:59.654527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.654490    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00"} err="failed to get container status \"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00\": rpc error: code = NotFound desc = could not find container \"4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00\": container with ID starting with 4a02193304d674f050261fc1eabcf9568df2029483e09c7423ba00b418267c00 not found: ID does not exist"
Nov 01 01:10:59.801056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.801019    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:10:59.804683 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:10:59.804662    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757"]
Nov 01 01:11:00.236792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:00.236760    4493 patch_prober.go:29] interesting pod/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:11:00.236971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:00.236835    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-718e19-openshift-default-7cf6dc8bdc-h6757" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:11:00.324015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:00.323970    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:11:00.501836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:00.501755    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d055b277-3003-4362-8457-2852717632b3" path="/var/lib/kubelet/pods/d055b277-3003-4362-8457-2852717632b3/volumes"
Nov 01 01:11:09.293820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.293757    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 01:11:09.449802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.449774    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\" (UniqueName: \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\") pod \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") "
Nov 01 01:11:09.449979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.449911    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\") pod \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\" (UID: \"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e\") "
Nov 01 01:11:09.450392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.450366    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume" pod "5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" (UID: "5fd366c6-e3f4-44c7-b6dc-9894ba1d705e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:09.452198 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.452165    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd" pod "5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" (UID: "5fd366c6-e3f4-44c7-b6dc-9894ba1d705e"). InnerVolumeSpecName "kube-api-access-clpzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:09.551519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.551434    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:09.551519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.551463    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpzd\" (UniqueName: \"kubernetes.io/projected/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e-kube-api-access-clpzd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:09.696716 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.696681    4493 generic.go:356] "Generic (PLEG): container finished" podID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerID="9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd" exitCode=0
Nov 01 01:11:09.696881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.696732    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"
Nov 01 01:11:09.696881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.696733    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" event={"ID":"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e","Type":"ContainerDied","Data":"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"}
Nov 01 01:11:09.696881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.696790    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk" event={"ID":"5fd366c6-e3f4-44c7-b6dc-9894ba1d705e","Type":"ContainerDied","Data":"a60eb5cf36eb3a26602ef950c6d14468bc7132616c7a06a594940e3279499e3d"}
Nov 01 01:11:09.696881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.696819    4493 scope.go:122] "RemoveContainer" containerID="9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"
Nov 01 01:11:09.706846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.706824    4493 scope.go:122] "RemoveContainer" containerID="9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"
Nov 01 01:11:09.707119 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:09.707096    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd\": container with ID starting with 9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd not found: ID does not exist" containerID="9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"
Nov 01 01:11:09.707191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.707127    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd"} err="failed to get container status \"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd\": rpc error: code = NotFound desc = could not find container \"9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd\": container with ID starting with 9b7e2dcc5c91553313dd2e10b8838f813034e6843d02cbd8aebd5f2f485845dd not found: ID does not exist"
Nov 01 01:11:09.865718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.865634    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 01:11:09.873430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:09.873407    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5a9c19-dbdc994d4-bkltk"]
Nov 01 01:11:10.501974 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:10.501934    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" path="/var/lib/kubelet/pods/5fd366c6-e3f4-44c7-b6dc-9894ba1d705e/volumes"
Nov 01 01:11:11.646969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.646909    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"]
Nov 01 01:11:11.647524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647505    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a9bfcc4a-8362-45cc-b943-b450b214d1df" containerName="payload-processing"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647526    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bfcc4a-8362-45cc-b943-b450b214d1df" containerName="payload-processing"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647540    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" containerName="maas-api"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647545    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" containerName="maas-api"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647557    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647565    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing"
Nov 01 01:11:11.647579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647576    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647581    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647589    4493 cpu_manager.go:414] "containerMap: removing container" podUID="00dc6468-96e7-47bd-8e46-2422ff75e562" containerName="maas-api"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647594    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dc6468-96e7-47bd-8e46-2422ff75e562" containerName="maas-api"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647609    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647614    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647620    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647624    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647633    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647640    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647656    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" containerName="payload-pre-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647663    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" containerName="payload-pre-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647676    4493 cpu_manager.go:414] "containerMap: removing container" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerName="payload-pre-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647680    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerName="payload-pre-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647688    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" containerName="payload-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647694    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" containerName="payload-processing"
Nov 01 01:11:11.647759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647761    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bfcc4a-8362-45cc-b943-b450b214d1df" containerName="payload-processing"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647769    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd366c6-e3f4-44c7-b6dc-9894ba1d705e" containerName="payload-processing"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647775    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff57b037-ade1-450a-a893-7d5f1c8bbe0c" containerName="maas-api"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647780    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c8c600-6f30-4c6b-bf55-281ae7918492" containerName="main"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647787    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dc6468-96e7-47bd-8e46-2422ff75e562" containerName="maas-api"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647794    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647800    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d132881-21f5-425f-b2c3-81af132525d2" containerName="main"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647805    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d055b277-3003-4362-8457-2852717632b3" containerName="istio-proxy"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647811    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba16e7ce-afd5-4491-97d7-40bd174e4f28" containerName="payload-pre-processing"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647816    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd55d2f0-a035-46dd-989f-cbc1063bc253" containerName="payload-pre-processing"
Nov 01 01:11:11.648288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.647820    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc099aae-5b39-494e-8a68-ec4b8ff3743e" containerName="payload-processing"
Nov 01 01:11:11.652267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.652244    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.710149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.710102    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"]
Nov 01 01:11:11.746880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.746833    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:11:11.747054 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.746892    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:11:11.775640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.775611    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.775640 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.775643    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7mz\" (UniqueName: \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.775889 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.775816    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.876743 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.876713    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.876919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.876750    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7mz\" (UniqueName: \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.876919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.876817    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.888684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.878179    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.889593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.889570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.942512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.942444    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7mz\" (UniqueName: \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\") pod \"maas-api-revoke-sub-bedf7164-zngbj\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:11.965862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:11.965835    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:12.351271 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:12.351245    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"]
Nov 01 01:11:12.352983 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:11:12.352957    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f68edb_6caf_4aaa_96aa_d4fdee6dc9cb.slice/crio-a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674 WatchSource:0}: Error finding container a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674: Status 404 returned error can't find the container with id a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674
Nov 01 01:11:12.716717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:12.716679    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj" event={"ID":"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb","Type":"ContainerStarted","Data":"ea9449b6477c297afc206fd3b7011fa3d4bd45b36dc75ce0e65875ab68a97d29"}
Nov 01 01:11:12.717157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:12.716738    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj" event={"ID":"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb","Type":"ContainerStarted","Data":"a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674"}
Nov 01 01:11:12.837249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:12.837193    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj" podStartSLOduration=1.837179377 podStartE2EDuration="1.837179377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:11:12.836428399 +0000 UTC m=+2862.514015338" watchObservedRunningTime="2026-11-01 01:11:12.837179377 +0000 UTC m=+2862.514766322"
Nov 01 01:11:13.725478 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:13.725447    4493 generic.go:356] "Generic (PLEG): container finished" podID="77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" containerID="ea9449b6477c297afc206fd3b7011fa3d4bd45b36dc75ce0e65875ab68a97d29" exitCode=0
Nov 01 01:11:13.725866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:13.725535    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj" event={"ID":"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb","Type":"ContainerDied","Data":"ea9449b6477c297afc206fd3b7011fa3d4bd45b36dc75ce0e65875ab68a97d29"}
Nov 01 01:11:14.843838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:14.843802    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:11:14.888055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:14.888027    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:14.992073 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:14.991989    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:11:15.016004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.015211    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\") pod \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") "
Nov 01 01:11:15.016004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.015294    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\" (UniqueName: \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\") pod \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") "
Nov 01 01:11:15.016004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.015398    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\") pod \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\" (UID: \"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb\") "
Nov 01 01:11:15.016004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.015597    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca" pod "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" (UID: "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:15.016004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.015815    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:15.017824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.017793    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz" pod "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" (UID: "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb"). InnerVolumeSpecName "kube-api-access-4d7mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:15.018182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.018158    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca" pod "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" (UID: "77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:11:15.117105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.117063    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d7mz\" (UniqueName: \"kubernetes.io/projected/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-kube-api-access-4d7mz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:15.117105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.117100    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:15.751784 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.751737    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj" event={"ID":"77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb","Type":"ContainerDied","Data":"a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674"}
Nov 01 01:11:15.752065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.751814    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9068c7f3791f392f02d77bfeadd5cb325f1c9aa4a58dbec5e094f4a1d351674"
Nov 01 01:11:15.752065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:15.751880    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"
Nov 01 01:11:17.092725 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.092691    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:11:17.093171 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.092919    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" podUID="d7f3c424-7472-4070-8fe3-2deb2592241f" containerName="maas-api" containerID="cri-o://3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4" gracePeriod=30
Nov 01 01:11:17.404688 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.404661    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:11:17.541634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.541604    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\") pod \"d7f3c424-7472-4070-8fe3-2deb2592241f\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") "
Nov 01 01:11:17.541786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.541662    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\") pod \"d7f3c424-7472-4070-8fe3-2deb2592241f\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") "
Nov 01 01:11:17.541786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.541699    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\") pod \"d7f3c424-7472-4070-8fe3-2deb2592241f\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") "
Nov 01 01:11:17.541978 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.541836    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\") pod \"d7f3c424-7472-4070-8fe3-2deb2592241f\" (UID: \"d7f3c424-7472-4070-8fe3-2deb2592241f\") "
Nov 01 01:11:17.542493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.542465    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca" pod "d7f3c424-7472-4070-8fe3-2deb2592241f" (UID: "d7f3c424-7472-4070-8fe3-2deb2592241f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:17.544446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.544415    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr" pod "d7f3c424-7472-4070-8fe3-2deb2592241f" (UID: "d7f3c424-7472-4070-8fe3-2deb2592241f"). InnerVolumeSpecName "kube-api-access-982lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:17.544575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.544517    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs" pod "d7f3c424-7472-4070-8fe3-2deb2592241f" (UID: "d7f3c424-7472-4070-8fe3-2deb2592241f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:11:17.544575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.544544    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls" pod "d7f3c424-7472-4070-8fe3-2deb2592241f" (UID: "d7f3c424-7472-4070-8fe3-2deb2592241f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:11:17.643255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.643174    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:17.643255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.643205    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3c424-7472-4070-8fe3-2deb2592241f-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:17.643255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.643214    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/d7f3c424-7472-4070-8fe3-2deb2592241f-kube-api-access-982lr\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:17.643255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.643224    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f3c424-7472-4070-8fe3-2deb2592241f-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:17.791157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.790315    4493 generic.go:356] "Generic (PLEG): container finished" podID="d7f3c424-7472-4070-8fe3-2deb2592241f" containerID="3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4" exitCode=0
Nov 01 01:11:17.791157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.790530    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" event={"ID":"d7f3c424-7472-4070-8fe3-2deb2592241f","Type":"ContainerDied","Data":"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"}
Nov 01 01:11:17.791157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.790583    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm" event={"ID":"d7f3c424-7472-4070-8fe3-2deb2592241f","Type":"ContainerDied","Data":"0aed875be65cda8f0621dd18ca8f1210da2773cb5c2e73e54dbfcd286dab0fea"}
Nov 01 01:11:17.791157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.790612    4493 scope.go:122] "RemoveContainer" containerID="3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"
Nov 01 01:11:17.791157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.790787    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"
Nov 01 01:11:17.808164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.808113    4493 scope.go:122] "RemoveContainer" containerID="3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"
Nov 01 01:11:17.808483 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:17.808460    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4\": container with ID starting with 3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4 not found: ID does not exist" containerID="3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"
Nov 01 01:11:17.808570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.808493    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4"} err="failed to get container status \"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4\": rpc error: code = NotFound desc = could not find container \"3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4\": container with ID starting with 3db629168daad13373ab1186266931bcdc4b35ef6aeb59d8524cfa69e6d102c4 not found: ID does not exist"
Nov 01 01:11:17.978954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:17.978912    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:11:18.028205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:18.028159    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9b61-5ddc957bc5-trtpm"]
Nov 01 01:11:18.502506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:18.502472    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f3c424-7472-4070-8fe3-2deb2592241f" path="/var/lib/kubelet/pods/d7f3c424-7472-4070-8fe3-2deb2592241f/volumes"
Nov 01 01:11:27.486275 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:27.486244    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:11:27.486848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:27.486495    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy" containerID="cri-o://573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc" gracePeriod=30
Nov 01 01:11:28.469290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469259    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"]
Nov 01 01:11:28.469710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469698    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d7f3c424-7472-4070-8fe3-2deb2592241f" containerName="maas-api"
Nov 01 01:11:28.469759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469711    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f3c424-7472-4070-8fe3-2deb2592241f" containerName="maas-api"
Nov 01 01:11:28.469759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469723    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" containerName="revoke-keys"
Nov 01 01:11:28.469759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469729    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" containerName="revoke-keys"
Nov 01 01:11:28.469851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469816    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" containerName="revoke-keys"
Nov 01 01:11:28.469851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.469834    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f3c424-7472-4070-8fe3-2deb2592241f" containerName="maas-api"
Nov 01 01:11:28.474462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.474445    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.480800 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.480768    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.480934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.480852    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsnv\" (UniqueName: \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.481005 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.480955    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.530428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.530399    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:11:28.580414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.580383    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:11:28.582007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.581984    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.582105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.582088    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsnv\" (UniqueName: \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.583340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.582169    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.583340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.583267    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.587796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.585869    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.593382 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.593349    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"]
Nov 01 01:11:28.644958 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.644932    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsnv\" (UniqueName: \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\") pod \"maas-api-revoke-keys-e2e-api-b-1402e8-mprr9\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:28.787905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:28.787823    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:29.169557 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:29.169487    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"]
Nov 01 01:11:29.171902 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:11:29.171876    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a29f773_84b0_4ada_8bde_0f059de86d34.slice/crio-681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8 WatchSource:0}: Error finding container 681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8: Status 404 returned error can't find the container with id 681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8
Nov 01 01:11:29.864114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:29.864078    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9" event={"ID":"3a29f773-84b0-4ada-8bde-0f059de86d34","Type":"ContainerStarted","Data":"27d046ae08c846d5b6692294b5f32b514075713a0d6fcfdfd427b47f806dac36"}
Nov 01 01:11:29.864576 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:29.864127    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9" event={"ID":"3a29f773-84b0-4ada-8bde-0f059de86d34","Type":"ContainerStarted","Data":"681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8"}
Nov 01 01:11:29.982339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:29.982280    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9" podStartSLOduration=1.9822624800000002 podStartE2EDuration="1.98226248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:11:29.980451593 +0000 UTC m=+2879.658038592" watchObservedRunningTime="2026-11-01 01:11:29.98226248 +0000 UTC m=+2879.659849426"
Nov 01 01:11:30.870758 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:30.870729    4493 generic.go:356] "Generic (PLEG): container finished" podID="3a29f773-84b0-4ada-8bde-0f059de86d34" containerID="27d046ae08c846d5b6692294b5f32b514075713a0d6fcfdfd427b47f806dac36" exitCode=0
Nov 01 01:11:30.871168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:30.870820    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9" event={"ID":"3a29f773-84b0-4ada-8bde-0f059de86d34","Type":"ContainerDied","Data":"27d046ae08c846d5b6692294b5f32b514075713a0d6fcfdfd427b47f806dac36"}
Nov 01 01:11:32.024102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.024079    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:32.119324 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.119289    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\") pod \"3a29f773-84b0-4ada-8bde-0f059de86d34\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") "
Nov 01 01:11:32.119506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.119333    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\") pod \"3a29f773-84b0-4ada-8bde-0f059de86d34\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") "
Nov 01 01:11:32.119506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.119393    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\" (UniqueName: \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\") pod \"3a29f773-84b0-4ada-8bde-0f059de86d34\" (UID: \"3a29f773-84b0-4ada-8bde-0f059de86d34\") "
Nov 01 01:11:32.119787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.119752    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca" pod "3a29f773-84b0-4ada-8bde-0f059de86d34" (UID: "3a29f773-84b0-4ada-8bde-0f059de86d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:32.121540 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.121515    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca" pod "3a29f773-84b0-4ada-8bde-0f059de86d34" (UID: "3a29f773-84b0-4ada-8bde-0f059de86d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:11:32.121677 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.121659    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv" pod "3a29f773-84b0-4ada-8bde-0f059de86d34" (UID: "3a29f773-84b0-4ada-8bde-0f059de86d34"). InnerVolumeSpecName "kube-api-access-tvsnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:32.221269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.221241    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsnv\" (UniqueName: \"kubernetes.io/projected/3a29f773-84b0-4ada-8bde-0f059de86d34-kube-api-access-tvsnv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.221269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.221270    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.221473 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.221284    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a29f773-84b0-4ada-8bde-0f059de86d34-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.744677 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.744650    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:11:32.828976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.828877    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.828976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.828941    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.828986    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829025    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829050    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829088    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829239    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829263    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:32.829439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829288    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829340    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\") pod \"298f865e-89c0-4fdd-9c04-2f3aed31619c\" (UID: \"298f865e-89c0-4fdd-9c04-2f3aed31619c\") "
Nov 01 01:11:32.829439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829358    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:32.829710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829487    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:32.829710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829565    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:32.829710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829693    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:32.829880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829824    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.829880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829836    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f865e-89c0-4fdd-9c04-2f3aed31619c-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.829880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829850    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.829880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829864    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.829880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.829875    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.831407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.831380    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:32.831629 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.831603    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "kube-api-access-7k8qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:32.831923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.831908    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:32.831985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.831935    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo" pod "298f865e-89c0-4fdd-9c04-2f3aed31619c" (UID: "298f865e-89c0-4fdd-9c04-2f3aed31619c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:11:32.883267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.883024    4493 generic.go:356] "Generic (PLEG): container finished" podID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerID="573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc" exitCode=0
Nov 01 01:11:32.883267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.883115    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerDied","Data":"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"}
Nov 01 01:11:32.883267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.883212    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn" event={"ID":"298f865e-89c0-4fdd-9c04-2f3aed31619c","Type":"ContainerDied","Data":"eaa1d4459731876581ca2cf1a36db1b6d2fd7dfcac4d7e715cd6ac4cba1f9a1f"}
Nov 01 01:11:32.883267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.883245    4493 scope.go:122] "RemoveContainer" containerID="573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"
Nov 01 01:11:32.883267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.883120    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"
Nov 01 01:11:32.902599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.902573    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"
Nov 01 01:11:32.902599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.902581    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9" event={"ID":"3a29f773-84b0-4ada-8bde-0f059de86d34","Type":"ContainerDied","Data":"681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8"}
Nov 01 01:11:32.902797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.902632    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="681968a61692dc4bb6bb4770e9b5b703f09c4089d0de73054f1ac29df1570ca8"
Nov 01 01:11:32.912601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.912576    4493 scope.go:122] "RemoveContainer" containerID="06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"
Nov 01 01:11:32.921864 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.921836    4493 scope.go:122] "RemoveContainer" containerID="573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"
Nov 01 01:11:32.922215 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:32.922185    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc\": container with ID starting with 573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc not found: ID does not exist" containerID="573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"
Nov 01 01:11:32.922290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.922227    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc"} err="failed to get container status \"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc\": rpc error: code = NotFound desc = could not find container \"573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc\": container with ID starting with 573a1c6cfe14a1552b53437ff3522dce899db01323dba2463e9a0351d494c4cc not found: ID does not exist"
Nov 01 01:11:32.922290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.922253    4493 scope.go:122] "RemoveContainer" containerID="06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"
Nov 01 01:11:32.922537 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:32.922517    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272\": container with ID starting with 06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272 not found: ID does not exist" containerID="06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"
Nov 01 01:11:32.922583 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.922543    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272"} err="failed to get container status \"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272\": rpc error: code = NotFound desc = could not find container \"06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272\": container with ID starting with 06d4e0f37a68eb6893f3a535a19207161100f58043abd887fbde4b0a20bd3272 not found: ID does not exist"
Nov 01 01:11:32.931221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.931201    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.931304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.931224    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.931304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.931235    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8qs\" (UniqueName: \"kubernetes.io/projected/298f865e-89c0-4fdd-9c04-2f3aed31619c-kube-api-access-7k8qs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:32.931304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:32.931246    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f865e-89c0-4fdd-9c04-2f3aed31619c-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:33.049571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:33.049487    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:11:33.052300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:33.052276    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6b9b61-openshift-default-7d949c44fc-g2pfn"]
Nov 01 01:11:34.502036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:34.502004    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" path="/var/lib/kubelet/pods/298f865e-89c0-4fdd-9c04-2f3aed31619c/volumes"
Nov 01 01:11:41.016906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:41.016872    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:11:41.128292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:41.128250    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:11:44.990747 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:44.990713    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:11:46.018049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.017990    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" podUID="0633dc74-e95f-47c5-ae58-16f58206db1f" containerName="payload-processing" containerID="cri-o://f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63" gracePeriod=25
Nov 01 01:11:46.103858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.103820    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 01:11:46.104066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.104042    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy" containerID="cri-o://26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd" gracePeriod=30
Nov 01 01:11:46.129418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.129377    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" podUID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" containerName="payload-pre-processing" containerID="cri-o://c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2" gracePeriod=25
Nov 01 01:11:46.287829 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.287802    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:11:46.385044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.385013    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:11:46.398426 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.398400    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\") pod \"0633dc74-e95f-47c5-ae58-16f58206db1f\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") "
Nov 01 01:11:46.398534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.398433    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\" (UniqueName: \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\") pod \"0633dc74-e95f-47c5-ae58-16f58206db1f\" (UID: \"0633dc74-e95f-47c5-ae58-16f58206db1f\") "
Nov 01 01:11:46.398784 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.398765    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume" pod "0633dc74-e95f-47c5-ae58-16f58206db1f" (UID: "0633dc74-e95f-47c5-ae58-16f58206db1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:46.400621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.400602    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g" pod "0633dc74-e95f-47c5-ae58-16f58206db1f" (UID: "0633dc74-e95f-47c5-ae58-16f58206db1f"). InnerVolumeSpecName "kube-api-access-2kr4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:46.499601 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.499570    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\") pod \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") "
Nov 01 01:11:46.499789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.499719    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\" (UniqueName: \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\") pod \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\" (UID: \"fb5eaba8-d92f-4826-82c0-2861f4bcf88a\") "
Nov 01 01:11:46.500006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.499984    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume" pod "fb5eaba8-d92f-4826-82c0-2861f4bcf88a" (UID: "fb5eaba8-d92f-4826-82c0-2861f4bcf88a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:46.500126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.500111    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0633dc74-e95f-47c5-ae58-16f58206db1f-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:46.500199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.500147    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kr4g\" (UniqueName: \"kubernetes.io/projected/0633dc74-e95f-47c5-ae58-16f58206db1f-kube-api-access-2kr4g\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:46.500199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.500162    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:46.502343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.502323    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj" pod "fb5eaba8-d92f-4826-82c0-2861f4bcf88a" (UID: "fb5eaba8-d92f-4826-82c0-2861f4bcf88a"). InnerVolumeSpecName "kube-api-access-t8qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:46.601466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.601381    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qpj\" (UniqueName: \"kubernetes.io/projected/fb5eaba8-d92f-4826-82c0-2861f4bcf88a-kube-api-access-t8qpj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:46.999417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.994560    4493 generic.go:356] "Generic (PLEG): container finished" podID="0633dc74-e95f-47c5-ae58-16f58206db1f" containerID="f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63" exitCode=0
Nov 01 01:11:46.999417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.994686    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" event={"ID":"0633dc74-e95f-47c5-ae58-16f58206db1f","Type":"ContainerDied","Data":"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"}
Nov 01 01:11:46.999417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.994745    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l" event={"ID":"0633dc74-e95f-47c5-ae58-16f58206db1f","Type":"ContainerDied","Data":"2a53b5245ae55ffd6abf88cd99c0ac6169db6546515403e4d43d7f2d99357ba0"}
Nov 01 01:11:46.999417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.994779    4493 scope.go:122] "RemoveContainer" containerID="f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"
Nov 01 01:11:46.999417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:46.994948    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"
Nov 01 01:11:47.005013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.003572    4493 generic.go:356] "Generic (PLEG): container finished" podID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" containerID="c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2" exitCode=0
Nov 01 01:11:47.005013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.003737    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" event={"ID":"fb5eaba8-d92f-4826-82c0-2861f4bcf88a","Type":"ContainerDied","Data":"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"}
Nov 01 01:11:47.005013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.003815    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d" event={"ID":"fb5eaba8-d92f-4826-82c0-2861f4bcf88a","Type":"ContainerDied","Data":"b6e774049ee3c6e1f6443981ed3c0331bdfedb436cd799260592ae4c71ad3c2a"}
Nov 01 01:11:47.005013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.003920    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"
Nov 01 01:11:47.042425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.041568    4493 scope.go:122] "RemoveContainer" containerID="f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"
Nov 01 01:11:47.042425 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:47.042007    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63\": container with ID starting with f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63 not found: ID does not exist" containerID="f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"
Nov 01 01:11:47.042425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.042048    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63"} err="failed to get container status \"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63\": rpc error: code = NotFound desc = could not find container \"f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63\": container with ID starting with f626e67fb6da44dbe9a908d8b0f4f8abd8e2a502a156839a1b51e8a65d2d0a63 not found: ID does not exist"
Nov 01 01:11:47.042425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.042073    4493 scope.go:122] "RemoveContainer" containerID="c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"
Nov 01 01:11:47.053370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.053348    4493 scope.go:122] "RemoveContainer" containerID="c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"
Nov 01 01:11:47.053709 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:47.053687    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2\": container with ID starting with c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2 not found: ID does not exist" containerID="c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"
Nov 01 01:11:47.053781 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.053723    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2"} err="failed to get container status \"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2\": rpc error: code = NotFound desc = could not find container \"c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2\": container with ID starting with c5636f03ad40ed9a2acdf24a3dc65c3c11bcc9de354710980123736db7404be2 not found: ID does not exist"
Nov 01 01:11:47.198844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.197817    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:11:47.210444 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.210407    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1402e8-5854d78c59-pst5d"]
Nov 01 01:11:47.358207 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.357993    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:11:47.360739 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:47.360710    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1402e8-7f44b9998d-xtl4l"]
Nov 01 01:11:48.502499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:48.502466    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0633dc74-e95f-47c5-ae58-16f58206db1f" path="/var/lib/kubelet/pods/0633dc74-e95f-47c5-ae58-16f58206db1f/volumes"
Nov 01 01:11:48.502883 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:48.502843    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" path="/var/lib/kubelet/pods/fb5eaba8-d92f-4826-82c0-2861f4bcf88a/volumes"
Nov 01 01:11:49.814005 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:49.813973    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:11:49.814487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:49.814291    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" containerID="cri-o://a2e862f656f62ca12a0621e4a345d4ad2401d65e00ba5d052dc1a6b2f9614b09" gracePeriod=30
Nov 01 01:11:51.436672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.436639    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:11:51.437301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437280    4493 cpu_manager.go:414] "containerMap: removing container" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:11:51.437301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437302    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437321    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" containerName="payload-pre-processing"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437328    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" containerName="payload-pre-processing"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437365    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0633dc74-e95f-47c5-ae58-16f58206db1f" containerName="payload-processing"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437374    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0633dc74-e95f-47c5-ae58-16f58206db1f" containerName="payload-processing"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437394    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3a29f773-84b0-4ada-8bde-0f059de86d34" containerName="revoke-keys"
Nov 01 01:11:51.437513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437402    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a29f773-84b0-4ada-8bde-0f059de86d34" containerName="revoke-keys"
Nov 01 01:11:51.437807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437526    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:11:51.437807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437537    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a29f773-84b0-4ada-8bde-0f059de86d34" containerName="revoke-keys"
Nov 01 01:11:51.437807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437554    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5eaba8-d92f-4826-82c0-2861f4bcf88a" containerName="payload-pre-processing"
Nov 01 01:11:51.437807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.437567    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0633dc74-e95f-47c5-ae58-16f58206db1f" containerName="payload-processing"
Nov 01 01:11:51.441262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.441241    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.447033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.447010    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 01:11:51.516421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.516393    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.558948    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559080    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559168    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559227    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559278    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559316    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559349    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559436    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559513    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\") pod \"ae34be36-7988-407b-a370-82f7cfb0d75e\" (UID: \"ae34be36-7988-407b-a370-82f7cfb0d75e\") "
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559694    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559767    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559828    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.559876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559865    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559983    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.560057    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.560087    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.560180    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.560210    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzvl\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559288    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559357    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:51.560934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.559644    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:51.561851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.561646    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:51.562284 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.562259    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:51.563071 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.563042    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:11:51.563376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.563346    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7fdf2c-openshift-default-dockercfg-dk7pr\""
Nov 01 01:11:51.565014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.564988    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:51.565102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.565052    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:51.565102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.565074    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6" pod "ae34be36-7988-407b-a370-82f7cfb0d75e" (UID: "ae34be36-7988-407b-a370-82f7cfb0d75e"). InnerVolumeSpecName "kube-api-access-gt6k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:51.661744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661660    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.661744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661709    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.661744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661728    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661748    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661901    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzvl\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.661994    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662055    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662109    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662176    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662188    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662337    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662365    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662388    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662499    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae34be36-7988-407b-a370-82f7cfb0d75e-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662513    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662524    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662534    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662542    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6k6\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-kube-api-access-gt6k6\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662553    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662567    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae34be36-7988-407b-a370-82f7cfb0d75e-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662576    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662582    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae34be36-7988-407b-a370-82f7cfb0d75e-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:51.662650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.662607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.664773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.664749    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.665027 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.664976    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.727046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.727009    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzvl\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.727495 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.727473    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:51.758334 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:51.758303    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:52.032961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.032924    4493 generic.go:356] "Generic (PLEG): container finished" podID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerID="26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd" exitCode=0
Nov 01 01:11:52.033126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.033001    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" event={"ID":"ae34be36-7988-407b-a370-82f7cfb0d75e","Type":"ContainerDied","Data":"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"}
Nov 01 01:11:52.033126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.033019    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"
Nov 01 01:11:52.033126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.033043    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285" event={"ID":"ae34be36-7988-407b-a370-82f7cfb0d75e","Type":"ContainerDied","Data":"355e53800ffc2bf1130b5484626ed7beac640e65cbe29c038fa9fef0b0cba7b8"}
Nov 01 01:11:52.033126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.033069    4493 scope.go:122] "RemoveContainer" containerID="26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"
Nov 01 01:11:52.044068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.044049    4493 scope.go:122] "RemoveContainer" containerID="26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"
Nov 01 01:11:52.044352 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:11:52.044333    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd\": container with ID starting with 26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd not found: ID does not exist" containerID="26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"
Nov 01 01:11:52.044392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.044361    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd"} err="failed to get container status \"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd\": rpc error: code = NotFound desc = could not find container \"26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd\": container with ID starting with 26de51e0e135ddac0ca89946880a6c9057af1b3670c78e83325ae1ae8361a7dd not found: ID does not exist"
Nov 01 01:11:52.198686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.198653    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:11:52.215066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.215023    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 01:11:52.221177 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.221148    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-bq285"]
Nov 01 01:11:52.232169 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:11:52.232120    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f6e54c_0449_4f99_bbe7_bf13739a9525.slice/crio-631ab8340219f37bb5829467a86c2e1da05cf13c70cb63f598be73ff8ce07987 WatchSource:0}: Error finding container 631ab8340219f37bb5829467a86c2e1da05cf13c70cb63f598be73ff8ce07987: Status 404 returned error can't find the container with id 631ab8340219f37bb5829467a86c2e1da05cf13c70cb63f598be73ff8ce07987
Nov 01 01:11:52.234558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.234525    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:11:52.234642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.234590    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:11:52.234642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.234618    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:11:52.503659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:52.503630    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" path="/var/lib/kubelet/pods/ae34be36-7988-407b-a370-82f7cfb0d75e/volumes"
Nov 01 01:11:53.040081 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:53.040047    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerStarted","Data":"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"}
Nov 01 01:11:53.040307 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:53.040102    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerStarted","Data":"631ab8340219f37bb5829467a86c2e1da05cf13c70cb63f598be73ff8ce07987"}
Nov 01 01:11:53.164957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:53.164901    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podStartSLOduration=2.164887746 podStartE2EDuration="2.164887746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:11:53.162397582 +0000 UTC m=+2902.839984526" watchObservedRunningTime="2026-11-01 01:11:53.164887746 +0000 UTC m=+2902.842474685"
Nov 01 01:11:53.759691 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:53.759362    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:53.766775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:53.766745    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:54.039878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:54.039793    4493 patch_prober.go:29] interesting pod/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:11:54.039878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:54.039862    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:11:54.046906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:54.046883    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:54.047736 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:54.047714    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:11:55.053672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.053647    4493 generic.go:356] "Generic (PLEG): container finished" podID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerID="a2e862f656f62ca12a0621e4a345d4ad2401d65e00ba5d052dc1a6b2f9614b09" exitCode=0
Nov 01 01:11:55.054053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.053712    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerDied","Data":"a2e862f656f62ca12a0621e4a345d4ad2401d65e00ba5d052dc1a6b2f9614b09"}
Nov 01 01:11:55.054053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.053771    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj" event={"ID":"19ede110-50e6-45fe-a56a-ad1bbeefe5d0","Type":"ContainerDied","Data":"0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad"}
Nov 01 01:11:55.054053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.053795    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0357578dc710dfa3eb64ba011579ab5d187922da14775d55c2f23f5ff83a83ad"
Nov 01 01:11:55.054053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.053812    4493 scope.go:122] "RemoveContainer" containerID="d40ab96fe59ebe104d1023e7b5735c7471e13e0b9b31b372c3bbd8c1fc650026"
Nov 01 01:11:55.070687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.070668    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:11:55.103338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103275    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103312    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103383    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103447    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103485    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103559    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:55.103694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103629    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103702    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:55.103789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103747    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103749    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:55.103876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103784    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.103974 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103853    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:11:55.103974 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103939    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:55.103974 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.103905    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\") pod \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\" (UID: \"19ede110-50e6-45fe-a56a-ad1bbeefe5d0\") "
Nov 01 01:11:55.104830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.104790    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.104830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.104824    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.105019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.104839    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.105019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.104852    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.105019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.104866    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.106154 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.106113    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:11:55.106237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.106120    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "kube-api-access-wsgmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:55.106299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.106263    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:11:55.106346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.106322    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy" pod "19ede110-50e6-45fe-a56a-ad1bbeefe5d0" (UID: "19ede110-50e6-45fe-a56a-ad1bbeefe5d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:11:55.205824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.205791    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgmj\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-kube-api-access-wsgmj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.205824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.205819    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.205824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.205828    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:55.206056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:55.205839    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ede110-50e6-45fe-a56a-ad1bbeefe5d0-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:11:56.061554 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:56.061521    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"
Nov 01 01:11:56.358864 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:56.358776    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:11:56.377730 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:56.377696    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-030da8-openshift-default-565b6f7bf9-5fwvj"]
Nov 01 01:11:56.501363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:11:56.501322    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" path="/var/lib/kubelet/pods/19ede110-50e6-45fe-a56a-ad1bbeefe5d0/volumes"
Nov 01 01:12:00.814489 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814456    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814859    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy"
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814868    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy"
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814884    4493 cpu_manager.go:414] "containerMap: removing container" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814889    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814912    4493 cpu_manager.go:414] "containerMap: removing container" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:12:00.814982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814918    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:12:00.815210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.814998    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.815210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.815006    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="298f865e-89c0-4fdd-9c04-2f3aed31619c" containerName="istio-proxy"
Nov 01 01:12:00.815210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.815018    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae34be36-7988-407b-a370-82f7cfb0d75e" containerName="istio-proxy"
Nov 01 01:12:00.818043 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.818026    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:00.869491 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.869454    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpsd\" (UniqueName: \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:00.869491 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.869493    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:00.870793 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.870769    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fdf2c\""
Nov 01 01:12:00.931211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.923118    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fdf2c-dockercfg-92vck\""
Nov 01 01:12:00.931528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.930989    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:12:00.963219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.963187    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:12:00.963723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.963703    4493 cpu_manager.go:414] "containerMap: removing container" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.963723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.963724    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.963894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.963840    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ede110-50e6-45fe-a56a-ad1bbeefe5d0" containerName="istio-proxy"
Nov 01 01:12:00.966894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.966873    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:00.970440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.970412    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpsd\" (UniqueName: \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:00.970574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.970470    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:00.971278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:00.971257    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:01.035253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.035219    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:12:01.038510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.038483    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpsd\" (UniqueName: \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\") pod \"payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:01.071266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.071188    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtrq\" (UniqueName: \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.071442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.071393    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.130860 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.130813    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:01.172534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.172446    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.172764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.172552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtrq\" (UniqueName: \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.174073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.174038    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.235036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.235005    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtrq\" (UniqueName: \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\") pod \"payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.280288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.280253    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:01.513057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.512975    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:12:01.515011 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:01.514969    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c1b70d_4bc2_40fd_9deb_1cf61c44f5b5.slice/crio-9c0e2c6fa87a99578627a6adfac69ad0ef8ffbac1d445166cf02a2bf6e8ea04d WatchSource:0}: Error finding container 9c0e2c6fa87a99578627a6adfac69ad0ef8ffbac1d445166cf02a2bf6e8ea04d: Status 404 returned error can't find the container with id 9c0e2c6fa87a99578627a6adfac69ad0ef8ffbac1d445166cf02a2bf6e8ea04d
Nov 01 01:12:01.515347 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.515329    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:12:01.661806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:01.661780    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:12:01.663790 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:01.663749    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aa87ef_2733_4911_9fc3_caf51fb5d669.slice/crio-a7237f408f50be56fdaf909c377976d1d9ad64a6ccf69f58ca1d0b8f064ce317 WatchSource:0}: Error finding container a7237f408f50be56fdaf909c377976d1d9ad64a6ccf69f58ca1d0b8f064ce317: Status 404 returned error can't find the container with id a7237f408f50be56fdaf909c377976d1d9ad64a6ccf69f58ca1d0b8f064ce317
Nov 01 01:12:02.093127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.093086    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" event={"ID":"93aa87ef-2733-4911-9fc3-caf51fb5d669","Type":"ContainerStarted","Data":"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"}
Nov 01 01:12:02.093649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.093183    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" event={"ID":"93aa87ef-2733-4911-9fc3-caf51fb5d669","Type":"ContainerStarted","Data":"a7237f408f50be56fdaf909c377976d1d9ad64a6ccf69f58ca1d0b8f064ce317"}
Nov 01 01:12:02.093649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.093315    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:02.094659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.094633    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" event={"ID":"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5","Type":"ContainerStarted","Data":"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"}
Nov 01 01:12:02.094764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.094680    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" event={"ID":"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5","Type":"ContainerStarted","Data":"9c0e2c6fa87a99578627a6adfac69ad0ef8ffbac1d445166cf02a2bf6e8ea04d"}
Nov 01 01:12:02.094764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.094709    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:02.218632 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:02.218545    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" podStartSLOduration=2.218524413 podStartE2EDuration="2.218524413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:02.216706136 +0000 UTC m=+2911.894293081" watchObservedRunningTime="2026-11-01 01:12:02.218524413 +0000 UTC m=+2911.896111359"
Nov 01 01:12:03.315328 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:03.315267    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" podStartSLOduration=3.315246444 podStartE2EDuration="3.315246444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:02.343570759 +0000 UTC m=+2912.021157706" watchObservedRunningTime="2026-11-01 01:12:03.315246444 +0000 UTC m=+2912.992833390"
Nov 01 01:12:03.316101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:03.316079    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:12:03.316393 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:03.316370    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" containerID="cri-o://3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f" gracePeriod=30
Nov 01 01:12:06.248168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:06.246592    4493 patch_prober.go:29] interesting pod/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:12:06.248168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:06.246659    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:12:08.575988 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.575957    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:12:08.666582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666487    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666544    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666576    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666686    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666765    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666819    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.666866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666843    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:08.666866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666858    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.667150 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666891    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.667150 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.666947    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:08.667150 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.667119    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:08.667320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.667299    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:12:08.667516 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.667496    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\") pod \"6d393838-b6da-4012-8830-fe231a36bfc9\" (UID: \"6d393838-b6da-4012-8830-fe231a36bfc9\") "
Nov 01 01:12:08.667908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.667878    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:08.668012 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.667997    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.668072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.668022    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.668072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.668036    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d393838-b6da-4012-8830-fe231a36bfc9-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.668072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.668049    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.668072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.668062    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.669169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.669117    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:08.669729 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.669707    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "kube-api-access-rvwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:08.669789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.669739    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:08.670418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.670398    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo" pod "6d393838-b6da-4012-8830-fe231a36bfc9" (UID: "6d393838-b6da-4012-8830-fe231a36bfc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:12:08.768915 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.768886    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.768915 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.768913    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d393838-b6da-4012-8830-fe231a36bfc9-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.769108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.768925    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwl8\" (UniqueName: \"kubernetes.io/projected/6d393838-b6da-4012-8830-fe231a36bfc9-kube-api-access-rvwl8\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.769108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.768936    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d393838-b6da-4012-8830-fe231a36bfc9-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:08.904369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:08.904338    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 01:12:09.137498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.137468    4493 generic.go:356] "Generic (PLEG): container finished" podID="6d393838-b6da-4012-8830-fe231a36bfc9" containerID="3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f" exitCode=0
Nov 01 01:12:09.137708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.137523    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"
Nov 01 01:12:09.137708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.137560    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerDied","Data":"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"}
Nov 01 01:12:09.137708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.137628    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9" event={"ID":"6d393838-b6da-4012-8830-fe231a36bfc9","Type":"ContainerDied","Data":"95baaca78149404df6c0d0853ce92fc3541c2a5dd3d6d7fa8dd24f3f9c617090"}
Nov 01 01:12:09.137708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.137658    4493 scope.go:122] "RemoveContainer" containerID="3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"
Nov 01 01:12:09.147374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.147342    4493 scope.go:122] "RemoveContainer" containerID="7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"
Nov 01 01:12:09.156325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.156306    4493 scope.go:122] "RemoveContainer" containerID="3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"
Nov 01 01:12:09.156599 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:09.156581    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f\": container with ID starting with 3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f not found: ID does not exist" containerID="3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"
Nov 01 01:12:09.156652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.156608    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f"} err="failed to get container status \"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f\": rpc error: code = NotFound desc = could not find container \"3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f\": container with ID starting with 3b8ee66373b5ef807b63b298b4142ff9d439c511858a13c1f4a9b2e462f5959f not found: ID does not exist"
Nov 01 01:12:09.156652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.156626    4493 scope.go:122] "RemoveContainer" containerID="7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"
Nov 01 01:12:09.156867 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:09.156850    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f\": container with ID starting with 7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f not found: ID does not exist" containerID="7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"
Nov 01 01:12:09.156925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.156877    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f"} err="failed to get container status \"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f\": rpc error: code = NotFound desc = could not find container \"7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f\": container with ID starting with 7edf956973bb71e22e12092c030632c33321f7b0b984018e7f0351a18ec4d31f not found: ID does not exist"
Nov 01 01:12:09.314615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.314584    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:12:09.325642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:09.325606    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cbea8a-openshift-default-79dcd58499-6mms9"]
Nov 01 01:12:10.501872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:10.501837    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" path="/var/lib/kubelet/pods/6d393838-b6da-4012-8830-fe231a36bfc9/volumes"
Nov 01 01:12:13.103033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:13.102959    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:12:13.103033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:13.103023    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:12:14.988594 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:14.988559    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:12:16.112419 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:16.112385    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 01:12:17.123087 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.123052    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:12:17.123597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.123578    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:17.123597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.123598    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:17.123719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.123708    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:17.128502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.128479    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.186845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.186797    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fdf2c-serving-cert\""
Nov 01 01:12:17.189392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.189362    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:12:17.265939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.265905    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.266098 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.265996    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.266196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.266177    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.266250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.266227    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8vv\" (UniqueName: \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.367517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.367478    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.367517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.367521    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8vv\" (UniqueName: \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.367792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.367605    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.367792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.367658    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.368400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.368326    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.370120 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.370099    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.370227 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.370152    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.432723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.432640    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8vv\" (UniqueName: \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\") pod \"maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.442626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.442601    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:17.711394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.711309    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"]
Nov 01 01:12:17.716322 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.716294    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1402e8-mprr9"]
Nov 01 01:12:17.836161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.836122    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:12:17.838242 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:17.838215    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebb6a93_3253_42a3_be62_fc124c12bb42.slice/crio-bdd4a2496cd5bf932f3a168a275e51de44a2b57319e8ec6cc04e3c7d207d6d56 WatchSource:0}: Error finding container bdd4a2496cd5bf932f3a168a275e51de44a2b57319e8ec6cc04e3c7d207d6d56: Status 404 returned error can't find the container with id bdd4a2496cd5bf932f3a168a275e51de44a2b57319e8ec6cc04e3c7d207d6d56
Nov 01 01:12:17.839721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:17.839703    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 01:12:18.186461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:18.186425    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" event={"ID":"7ebb6a93-3253-42a3-be62-fc124c12bb42","Type":"ContainerStarted","Data":"bdd4a2496cd5bf932f3a168a275e51de44a2b57319e8ec6cc04e3c7d207d6d56"}
Nov 01 01:12:18.501194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:18.501159    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a29f773-84b0-4ada-8bde-0f059de86d34" path="/var/lib/kubelet/pods/3a29f773-84b0-4ada-8bde-0f059de86d34/volumes"
Nov 01 01:12:19.196316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:19.196224    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" event={"ID":"7ebb6a93-3253-42a3-be62-fc124c12bb42","Type":"ContainerStarted","Data":"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"}
Nov 01 01:12:19.196687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:19.196420    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:19.318926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:19.318865    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" podStartSLOduration=1.27527928 podStartE2EDuration="2.318847966s" totalImagesPullingTime="1.043568686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:19.318401662 +0000 UTC m=+2928.995988607" watchObservedRunningTime="2026-11-01 01:12:19.318847966 +0000 UTC m=+2928.996434911"
Nov 01 01:12:21.112791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:21.112727    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing" containerID="cri-o://3f5e55663f567735818d7b99478a7e9eefe8bbe0e9e7d97c135441e3aadd6167" gracePeriod=25
Nov 01 01:12:21.262639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:21.262603    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:12:23.925458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:23.925411    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main" containerID="cri-o://05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13" gracePeriod=45
Nov 01 01:12:24.180444 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.180377    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 01:12:24.225836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.225802    4493 generic.go:356] "Generic (PLEG): container finished" podID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerID="05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13" exitCode=0
Nov 01 01:12:24.226006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.225856    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"
Nov 01 01:12:24.226006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.225892    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" event={"ID":"b60ab262-9c75-487e-9031-7a0fa02e387c","Type":"ContainerDied","Data":"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"}
Nov 01 01:12:24.226006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.225937    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm" event={"ID":"b60ab262-9c75-487e-9031-7a0fa02e387c","Type":"ContainerDied","Data":"8070b18c9d370b63f4c3793987ba5259d92d8837b8b5ace1c39fb7e8d97f398e"}
Nov 01 01:12:24.226006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.225957    4493 scope.go:122] "RemoveContainer" containerID="05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"
Nov 01 01:12:24.234810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.234791    4493 scope.go:122] "RemoveContainer" containerID="05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"
Nov 01 01:12:24.235089 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:24.235070    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13\": container with ID starting with 05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13 not found: ID does not exist" containerID="05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"
Nov 01 01:12:24.235169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.235098    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13"} err="failed to get container status \"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13\": rpc error: code = NotFound desc = could not find container \"05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13\": container with ID starting with 05a1e1218741b741f6c32c20cd0e3e8da9ef6ad9851f95da4431150e4df24a13 not found: ID does not exist"
Nov 01 01:12:24.248057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248029    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248066    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\" (UniqueName: \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248090    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248161    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248218    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248322    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\") pod \"b60ab262-9c75-487e-9031-7a0fa02e387c\" (UID: \"b60ab262-9c75-487e-9031-7a0fa02e387c\") "
Nov 01 01:12:24.248593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248339    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:24.248593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248428    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:24.248593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248513    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:24.249021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248762    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.249021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248780    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.249021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.248794    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.250349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.250325    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:24.250430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.250414    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:12:24.250504 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.250489    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl" pod "b60ab262-9c75-487e-9031-7a0fa02e387c" (UID: "b60ab262-9c75-487e-9031-7a0fa02e387c"). InnerVolumeSpecName "kube-api-access-xfbpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:24.350270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.350232    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfbpl\" (UniqueName: \"kubernetes.io/projected/b60ab262-9c75-487e-9031-7a0fa02e387c-kube-api-access-xfbpl\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.350270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.350261    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60ab262-9c75-487e-9031-7a0fa02e387c-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.350270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.350271    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60ab262-9c75-487e-9031-7a0fa02e387c-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:24.658566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.658537    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 01:12:24.663682 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.663652    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/premium-w1-30fb19-kserve-6c8874ffd6-txsbm"]
Nov 01 01:12:24.680247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.680220    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:12:24.680448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:24.680426    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy" containerID="cri-o://0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c" gracePeriod=30
Nov 01 01:12:25.206990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:25.206957    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:12:26.501889 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:26.501859    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" path="/var/lib/kubelet/pods/b60ab262-9c75-487e-9031-7a0fa02e387c/volumes"
Nov 01 01:12:29.954256 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:29.954194    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:12:30.011589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011552    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.011741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011665    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.011741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011694    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.011741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011730    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.011915 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011765    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.011915 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011837    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.012017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011924    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.012017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011954    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.012017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.011988    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\") pod \"8f72f912-d6b2-41d9-a4b1-64c582020a36\" (UID: \"8f72f912-d6b2-41d9-a4b1-64c582020a36\") "
Nov 01 01:12:30.012213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012068    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:30.012213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012110    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:30.012502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012399    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:30.012502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012407    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:12:30.012502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012474    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:30.012888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012458    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.012888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.012630    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.014188 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.014156    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:12:30.014302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.014261    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "kube-api-access-6x2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:30.014378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.014294    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:30.014460 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.014442    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy" pod "8f72f912-d6b2-41d9-a4b1-64c582020a36" (UID: "8f72f912-d6b2-41d9-a4b1-64c582020a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:30.113723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113694    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113722    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113731    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113740    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113748    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72f912-d6b2-41d9-a4b1-64c582020a36-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113757    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72f912-d6b2-41d9-a4b1-64c582020a36-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.113966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.113765    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2sv\" (UniqueName: \"kubernetes.io/projected/8f72f912-d6b2-41d9-a4b1-64c582020a36-kube-api-access-6x2sv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:30.183780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.183744    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 01:12:30.259602 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.259574    4493 generic.go:356] "Generic (PLEG): container finished" podID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerID="0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c" exitCode=0
Nov 01 01:12:30.259772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.259628    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"
Nov 01 01:12:30.259772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.259643    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerDied","Data":"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"}
Nov 01 01:12:30.259772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.259686    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm" event={"ID":"8f72f912-d6b2-41d9-a4b1-64c582020a36","Type":"ContainerDied","Data":"d2b69debcfb6ff63e1b3ef440614b09cf6936709af38ae557592820910583c57"}
Nov 01 01:12:30.259772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.259708    4493 scope.go:122] "RemoveContainer" containerID="0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"
Nov 01 01:12:30.269507 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.269481    4493 scope.go:122] "RemoveContainer" containerID="21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"
Nov 01 01:12:30.277815 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.277796    4493 scope.go:122] "RemoveContainer" containerID="0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"
Nov 01 01:12:30.278108 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:30.278090    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c\": container with ID starting with 0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c not found: ID does not exist" containerID="0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"
Nov 01 01:12:30.278267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.278117    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c"} err="failed to get container status \"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c\": rpc error: code = NotFound desc = could not find container \"0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c\": container with ID starting with 0529bebad93e7ab82aa3b9b091a36f55d79ae9510f8d4e0c990d2b53e538523c not found: ID does not exist"
Nov 01 01:12:30.278267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.278157    4493 scope.go:122] "RemoveContainer" containerID="21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"
Nov 01 01:12:30.278428 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:30.278413    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db\": container with ID starting with 21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db not found: ID does not exist" containerID="21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"
Nov 01 01:12:30.278472 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.278436    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db"} err="failed to get container status \"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db\": rpc error: code = NotFound desc = could not find container \"21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db\": container with ID starting with 21372cd1c337a86d83ab349175ec7f3193075c5a3cb3694fa9284dc53c84e7db not found: ID does not exist"
Nov 01 01:12:30.436045 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.436005    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:12:30.436246 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.436068    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1402e8-openshift-default-644849d5ff-s46wm"]
Nov 01 01:12:30.501649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:30.501623    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" path="/var/lib/kubelet/pods/8f72f912-d6b2-41d9-a4b1-64c582020a36/volumes"
Nov 01 01:12:31.089393 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:31.089364    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:12:31.262767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:31.262725    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:12:36.867378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867332    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867752    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867763    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867775    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867780    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867795    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867801    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867815    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.867862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867819    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.868119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867882    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d393838-b6da-4012-8830-fe231a36bfc9" containerName="istio-proxy"
Nov 01 01:12:36.868119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867891    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.868119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867902    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60ab262-9c75-487e-9031-7a0fa02e387c" containerName="main"
Nov 01 01:12:36.868119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.867907    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f72f912-d6b2-41d9-a4b1-64c582020a36" containerName="istio-proxy"
Nov 01 01:12:36.872633 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:36.872612    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004174 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004127    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004188    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004359    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004409    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004524    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004611 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004596    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004630    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004664    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd2f8\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.004764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.004717    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.079741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.079706    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:12:37.105978 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.105942    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd2f8\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.105978 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.105981    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106027    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106065    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106177    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106259    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106323    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106352    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106685    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106744    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.106872 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.106819    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.107096 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.107072    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.107419 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.107394    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.109115 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.109088    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.109402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.109380    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.167179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.167075    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.167179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.167161    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd2f8\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.186010 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.185983    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:37.622765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.622728    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:12:37.623685 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:37.623659    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fd31e3_1bd6_4bd5_9a66_909108206b20.slice/crio-b20866c070ec80fb2126ab8f50afafe0d6137f9c61ed81fd5ddbf7ff26c88942 WatchSource:0}: Error finding container b20866c070ec80fb2126ab8f50afafe0d6137f9c61ed81fd5ddbf7ff26c88942: Status 404 returned error can't find the container with id b20866c070ec80fb2126ab8f50afafe0d6137f9c61ed81fd5ddbf7ff26c88942
Nov 01 01:12:37.627213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.627182    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:37.627335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.627252    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:37.627335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:37.627280    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:38.305346 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:38.305311    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" event={"ID":"c0fd31e3-1bd6-4bd5-9a66-909108206b20","Type":"ContainerStarted","Data":"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"}
Nov 01 01:12:38.305841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:38.305380    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" event={"ID":"c0fd31e3-1bd6-4bd5-9a66-909108206b20","Type":"ContainerStarted","Data":"b20866c070ec80fb2126ab8f50afafe0d6137f9c61ed81fd5ddbf7ff26c88942"}
Nov 01 01:12:39.186214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:39.186171    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:39.187597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:39.187565    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 01:12:39.187725 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:39.187618    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 01:12:40.089276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:40.089232    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:12:40.089678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:40.089291    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:12:40.187508 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:40.187471    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 01:12:40.187685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:40.187523    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 01:12:41.186874 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.186843    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 01:12:41.187252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.186900    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 01:12:41.262297 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:41.262242    4493 logging.go:55] [core] [Channel #5962 SubChannel #5963]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: connect: connection refused"
Nov 01 01:12:41.349593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.349551    4493 generic.go:356] "Generic (PLEG): container finished" podID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerID="3f5e55663f567735818d7b99478a7e9eefe8bbe0e9e7d97c135441e3aadd6167" exitCode=0
Nov 01 01:12:41.349786 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.349655    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" event={"ID":"0f67a0cc-b273-4635-81e9-98fcb4da881c","Type":"ContainerDied","Data":"3f5e55663f567735818d7b99478a7e9eefe8bbe0e9e7d97c135441e3aadd6167"}
Nov 01 01:12:41.351886 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.351859    4493 generic.go:356] "Generic (PLEG): container finished" podID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerID="d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf" exitCode=0
Nov 01 01:12:41.352050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.352026    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerDied","Data":"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"}
Nov 01 01:12:41.352523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.352490    4493 scope.go:122] "RemoveContainer" containerID="d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"
Nov 01 01:12:41.360463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.360421    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:41.360591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.360502    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:41.360591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.360544    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:12:41.428411 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.424478    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 01:12:41.563655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.563597    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podStartSLOduration=5.563580797 podStartE2EDuration="5.563580797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:38.435576421 +0000 UTC m=+2948.113163366" watchObservedRunningTime="2026-11-01 01:12:41.563580797 +0000 UTC m=+2951.241167743"
Nov 01 01:12:41.586173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.586116    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\" (UniqueName: \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\") pod \"0f67a0cc-b273-4635-81e9-98fcb4da881c\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") "
Nov 01 01:12:41.586366 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.586349    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\") pod \"0f67a0cc-b273-4635-81e9-98fcb4da881c\" (UID: \"0f67a0cc-b273-4635-81e9-98fcb4da881c\") "
Nov 01 01:12:41.586759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.586733    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume" pod "0f67a0cc-b273-4635-81e9-98fcb4da881c" (UID: "0f67a0cc-b273-4635-81e9-98fcb4da881c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:12:41.588483 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.588455    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz" pod "0f67a0cc-b273-4635-81e9-98fcb4da881c" (UID: "0f67a0cc-b273-4635-81e9-98fcb4da881c"). InnerVolumeSpecName "kube-api-access-8gjbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:41.688896 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.688784    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67a0cc-b273-4635-81e9-98fcb4da881c-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:41.688896 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:41.688818    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjbz\" (UniqueName: \"kubernetes.io/projected/0f67a0cc-b273-4635-81e9-98fcb4da881c-kube-api-access-8gjbz\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:42.187429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.187391    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 01:12:42.187840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.187449    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 01:12:42.264591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.264543    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.44:50052\" within 1s: context deadline exceeded"
Nov 01 01:12:42.264902 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.264885    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 01:12:42.265166 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:12:42.265123    4493 logging.go:55] [core] [Channel #5962 SubChannel #5963]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: operation was canceled"
Nov 01 01:12:42.382326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.382296    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"
Nov 01 01:12:42.386698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.386661    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq" event={"ID":"0f67a0cc-b273-4635-81e9-98fcb4da881c","Type":"ContainerDied","Data":"d2a12f73a82d434bf4c25b655861d5a89cd94cb6c2f0108c9bf42e36976753e9"}
Nov 01 01:12:42.386947 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.386933    4493 scope.go:122] "RemoveContainer" containerID="3f5e55663f567735818d7b99478a7e9eefe8bbe0e9e7d97c135441e3aadd6167"
Nov 01 01:12:42.398235 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.398204    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerStarted","Data":"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"}
Nov 01 01:12:42.697046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.697016    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 01:12:42.700488 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.700462    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-30fb19-67d5dfcd66-drnxq"]
Nov 01 01:12:42.766215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.762544    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:12:42.766215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.762677    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 01:12:42.766215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:42.762730    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 01:12:43.187682 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:43.187598    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 01:12:43.188275 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:43.188245    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 01:12:43.760595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:43.760556    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 01:12:43.760922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:43.760873    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 01:12:44.404251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.404214    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:44.404741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.404644    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:44.405629 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.405599    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:12:44.509035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.508998    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" path="/var/lib/kubelet/pods/0f67a0cc-b273-4635-81e9-98fcb4da881c/volumes"
Nov 01 01:12:44.759077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.759039    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 01:12:44.759290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:44.759109    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 01:12:44.989379 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:12:44.989346    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:12:45.205312 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.204245    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" containerID="cri-o://b4722915f64b2ebdfddef77257c77a458c8bd67313266bb7bcb55b3baf585ccd" gracePeriod=45
Nov 01 01:12:45.442998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.442963    4493 generic.go:356] "Generic (PLEG): container finished" podID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerID="b4722915f64b2ebdfddef77257c77a458c8bd67313266bb7bcb55b3baf585ccd" exitCode=0
Nov 01 01:12:45.444041 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.444000    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" event={"ID":"ac707d00-4683-4bc6-a202-4d6e69adc8cf","Type":"ContainerDied","Data":"b4722915f64b2ebdfddef77257c77a458c8bd67313266bb7bcb55b3baf585ccd"}
Nov 01 01:12:45.502105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.502081    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 01:12:45.659428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659399    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.659635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659457    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\" (UniqueName: \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.659635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659505    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.659635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659535    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.659635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659623    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.659841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.659780    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\") pod \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\" (UID: \"ac707d00-4683-4bc6-a202-4d6e69adc8cf\") "
Nov 01 01:12:45.660225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660106    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:45.660225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660127    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:45.660455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660247    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:45.660741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660716    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:45.660844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660743    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:45.660844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.660757    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:45.662975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.662946    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "kube-api-access-dnl8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:12:45.665517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.665497    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:12:45.665616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.665574    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs" pod "ac707d00-4683-4bc6-a202-4d6e69adc8cf" (UID: "ac707d00-4683-4bc6-a202-4d6e69adc8cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:12:45.759589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.759497    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 01:12:45.759589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.759555    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 01:12:45.762271 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.762243    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac707d00-4683-4bc6-a202-4d6e69adc8cf-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:45.762384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.762276    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnl8c\" (UniqueName: \"kubernetes.io/projected/ac707d00-4683-4bc6-a202-4d6e69adc8cf-kube-api-access-dnl8c\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:45.762384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:45.762292    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac707d00-4683-4bc6-a202-4d6e69adc8cf-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:12:46.450904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.450869    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"
Nov 01 01:12:46.450904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.450901    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" event={"ID":"ac707d00-4683-4bc6-a202-4d6e69adc8cf","Type":"ContainerDied","Data":"0968711c4e1ce83447c1d58badcc15cb0a47cb99b27544b6b5f45e9bb1685ea9"}
Nov 01 01:12:46.451479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.450948    4493 scope.go:122] "RemoveContainer" containerID="b4722915f64b2ebdfddef77257c77a458c8bd67313266bb7bcb55b3baf585ccd"
Nov 01 01:12:46.497743 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.497282    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:12:46.642298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.638939    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 01:12:46.644627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.644600    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-w6-22d7b2-kserve-78c9744cc4-ls5bm"]
Nov 01 01:12:46.758750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.758712    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 01:12:46.758954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:46.758770    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 01:12:47.881773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:47.881732    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:12:47.882713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:47.882693    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:12:47.883742 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:47.883724    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:12:48.501780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:48.501749    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" path="/var/lib/kubelet/pods/ac707d00-4683-4bc6-a202-4d6e69adc8cf/volumes"
Nov 01 01:12:55.427371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:55.427321    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:12:57.973741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:57.973701    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 01:12:57.974213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:12:57.973899    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" containerID="cri-o://dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7" gracePeriod=30
Nov 01 01:13:00.994035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:00.993994    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:13:00.994443 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:00.994054    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:13:03.228052 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.228019    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 01:13:03.275348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275267    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275376    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275411    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275460    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275520    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275569    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275610    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275671    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275709    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\") pod \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\" (UID: \"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c\") "
Nov 01 01:13:03.275887 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275834    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:13:03.276019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.275978    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:13:03.276182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.276160    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:13:03.276256 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.276176    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:13:03.276298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.276253    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.276298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.276273    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.276298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.276276    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:13:03.278350 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.278322    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:13:03.278438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.278332    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:13:03.278505 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.278465    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:13:03.278616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.278556    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5" pod "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" (UID: "bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c"). InnerVolumeSpecName "kube-api-access-thmb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:13:03.377000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.376958    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.376997    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.377007    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.377016    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmb5\" (UniqueName: \"kubernetes.io/projected/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-kube-api-access-thmb5\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.377025    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.377035    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.377266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.377043    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:03.589807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.589733    4493 generic.go:356] "Generic (PLEG): container finished" podID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerID="dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7" exitCode=0
Nov 01 01:13:03.589807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.589783    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"
Nov 01 01:13:03.590016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.589814    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerDied","Data":"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"}
Nov 01 01:13:03.590016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.589867    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv" event={"ID":"bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c","Type":"ContainerDied","Data":"e46bd1d24684fd3d79c0e7d2d4b462117e83c2854282308077e9d0b591171e32"}
Nov 01 01:13:03.590016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.589889    4493 scope.go:122] "RemoveContainer" containerID="dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"
Nov 01 01:13:03.600120 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.600100    4493 scope.go:122] "RemoveContainer" containerID="055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"
Nov 01 01:13:03.608492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.608474    4493 scope.go:122] "RemoveContainer" containerID="dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"
Nov 01 01:13:03.608806 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:13:03.608787    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7\": container with ID starting with dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7 not found: ID does not exist" containerID="dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"
Nov 01 01:13:03.608884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.608818    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7"} err="failed to get container status \"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7\": rpc error: code = NotFound desc = could not find container \"dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7\": container with ID starting with dae06404cf8c64a1f973d6aa4a2c1454ba53356814bdd1311f965ee379b264c7 not found: ID does not exist"
Nov 01 01:13:03.608884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.608845    4493 scope.go:122] "RemoveContainer" containerID="055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"
Nov 01 01:13:03.609123 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:13:03.609106    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62\": container with ID starting with 055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62 not found: ID does not exist" containerID="055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"
Nov 01 01:13:03.609195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.609157    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62"} err="failed to get container status \"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62\": rpc error: code = NotFound desc = could not find container \"055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62\": container with ID starting with 055d7699025db00640156dcc7293160a5f59835fa6e1e49a036a05f5649dcd62 not found: ID does not exist"
Nov 01 01:13:03.761269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.761236    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 01:13:03.765400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:03.765379    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-lmgnv"]
Nov 01 01:13:04.501967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:04.501931    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" path="/var/lib/kubelet/pods/bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c/volumes"
Nov 01 01:13:14.150028 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:14.149992    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:13:15.007223 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:13:15.007186    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:13:19.098472 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.098434    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 01:13:19.098960 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.098645    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" podUID="cb271338-c36e-44a4-93ed-08a1a9195a74" containerName="maas-api" containerID="cri-o://e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf" gracePeriod=30
Nov 01 01:13:19.371725 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.371701    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 01:13:19.442479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.442441    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\") pod \"cb271338-c36e-44a4-93ed-08a1a9195a74\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") "
Nov 01 01:13:19.442676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.442602    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\" (UniqueName: \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\") pod \"cb271338-c36e-44a4-93ed-08a1a9195a74\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") "
Nov 01 01:13:19.442676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.442650    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") pod \"cb271338-c36e-44a4-93ed-08a1a9195a74\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") "
Nov 01 01:13:19.442797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.442703    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\") pod \"cb271338-c36e-44a4-93ed-08a1a9195a74\" (UID: \"cb271338-c36e-44a4-93ed-08a1a9195a74\") "
Nov 01 01:13:19.442903 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.442870    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca" pod "cb271338-c36e-44a4-93ed-08a1a9195a74" (UID: "cb271338-c36e-44a4-93ed-08a1a9195a74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:13:19.443044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.443020    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb271338-c36e-44a4-93ed-08a1a9195a74-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:19.444986 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.444955    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls" pod "cb271338-c36e-44a4-93ed-08a1a9195a74" (UID: "cb271338-c36e-44a4-93ed-08a1a9195a74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:13:19.444986 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.444980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs" pod "cb271338-c36e-44a4-93ed-08a1a9195a74" (UID: "cb271338-c36e-44a4-93ed-08a1a9195a74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:13:19.445125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.445030    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7" pod "cb271338-c36e-44a4-93ed-08a1a9195a74" (UID: "cb271338-c36e-44a4-93ed-08a1a9195a74"). InnerVolumeSpecName "kube-api-access-s87b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:13:19.544606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.544571    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s87b7\" (UniqueName: \"kubernetes.io/projected/cb271338-c36e-44a4-93ed-08a1a9195a74-kube-api-access-s87b7\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:19.544606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.544600    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:19.544606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.544611    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb271338-c36e-44a4-93ed-08a1a9195a74-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:13:19.678650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.678569    4493 generic.go:356] "Generic (PLEG): container finished" podID="cb271338-c36e-44a4-93ed-08a1a9195a74" containerID="e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf" exitCode=0
Nov 01 01:13:19.678650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.678604    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" event={"ID":"cb271338-c36e-44a4-93ed-08a1a9195a74","Type":"ContainerDied","Data":"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"}
Nov 01 01:13:19.678650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.678618    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"
Nov 01 01:13:19.678650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.678644    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl" event={"ID":"cb271338-c36e-44a4-93ed-08a1a9195a74","Type":"ContainerDied","Data":"8742f32ec4833c3c0ba64604a004e4e09bf86bde594bbefda620ee46637a8c3f"}
Nov 01 01:13:19.678946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.678665    4493 scope.go:122] "RemoveContainer" containerID="e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"
Nov 01 01:13:19.688855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.688837    4493 scope.go:122] "RemoveContainer" containerID="e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"
Nov 01 01:13:19.689195 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:13:19.689168    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf\": container with ID starting with e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf not found: ID does not exist" containerID="e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"
Nov 01 01:13:19.689300 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.689204    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf"} err="failed to get container status \"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf\": rpc error: code = NotFound desc = could not find container \"e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf\": container with ID starting with e1cc31d26a008aa08d9b73ade8902cb22e226c09a491c07aa1fae1fdca41cfcf not found: ID does not exist"
Nov 01 01:13:19.843235 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.843208    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 01:13:19.846592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:19.846568    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-22d7b2-6c9594d846-rp9kl"]
Nov 01 01:13:20.502989 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:20.502953    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb271338-c36e-44a4-93ed-08a1a9195a74" path="/var/lib/kubelet/pods/cb271338-c36e-44a4-93ed-08a1a9195a74/volumes"
Nov 01 01:13:35.198400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:35.198366    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:13:44.227211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227116    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:13:44.227615 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227599    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227618    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227628    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cb271338-c36e-44a4-93ed-08a1a9195a74" containerName="maas-api"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227633    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb271338-c36e-44a4-93ed-08a1a9195a74" containerName="maas-api"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227642    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227647    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227653    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227658    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227671    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227677    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227736    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227743    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb271338-c36e-44a4-93ed-08a1a9195a74" containerName="maas-api"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227751    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac707d00-4683-4bc6-a202-4d6e69adc8cf" containerName="main"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227757    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3d0a61-b39e-4f6d-8d12-db11cfdfdc8c" containerName="istio-proxy"
Nov 01 01:13:44.227893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.227764    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f67a0cc-b273-4635-81e9-98fcb4da881c" containerName="payload-processing"
Nov 01 01:13:44.232362 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.232342    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.285075 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.285034    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7fdf2c\"/\"default-dockercfg-z9rnd\""
Nov 01 01:13:44.331959 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.331929    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.332141 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.331965    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.332141 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.332068    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.332249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.332201    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9pp\" (UniqueName: \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.332249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.332244    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.332313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.332259    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.335156 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.335127    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7fdf2c\"/\"openshift-service-ca.crt\""
Nov 01 01:13:44.335279 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.335265    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7fdf2c\"/\"test-model-7fdf2c-kserve-self-signed-certs\""
Nov 01 01:13:44.337290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.337272    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:13:44.337422 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.337409    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7fdf2c\"/\"kube-root-ca.crt\""
Nov 01 01:13:44.432739 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432703    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.432935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432777    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9pp\" (UniqueName: \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.432935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432806    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.432935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432821    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.432935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432854    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.432935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.432874    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.433237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.433181    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.433276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.433233    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.433314 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.433300    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.435192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.435174    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.435481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.435465    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.494070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.494000    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9pp\" (UniqueName: \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\") pod \"test-model-7fdf2c-kserve-554dc78bf5-jrw8m\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") " pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.545264 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.545235    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:44.921087 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:44.921060    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:13:44.923009 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:13:44.922983    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b2eafc_cbc7_4d73_a440_be20e3c19283.slice/crio-a98c94677fbd018f9743ec2660479a495907de05377521db525984fdc15688b1 WatchSource:0}: Error finding container a98c94677fbd018f9743ec2660479a495907de05377521db525984fdc15688b1: Status 404 returned error can't find the container with id a98c94677fbd018f9743ec2660479a495907de05377521db525984fdc15688b1
Nov 01 01:13:44.991053 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:13:44.991016    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:13:45.804391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:45.804358    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" event={"ID":"36b2eafc-cbc7-4d73-a440-be20e3c19283","Type":"ContainerStarted","Data":"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"}
Nov 01 01:13:45.804881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:45.804404    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" event={"ID":"36b2eafc-cbc7-4d73-a440-be20e3c19283","Type":"ContainerStarted","Data":"a98c94677fbd018f9743ec2660479a495907de05377521db525984fdc15688b1"}
Nov 01 01:13:46.545718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:46.545684    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:48.007011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:48.006951    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" podStartSLOduration=4.00692956 podStartE2EDuration="4.00692956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:13:45.928436411 +0000 UTC m=+3015.606023387" watchObservedRunningTime="2026-11-01 01:13:48.00692956 +0000 UTC m=+3017.684516508"
Nov 01 01:13:48.008823 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:48.008800    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:13:54.546289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:54.546254    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:54.558898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:54.558869    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:13:54.863343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:13:54.863261    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:14:14.989693 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:14:14.989655    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:14:15.826394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:14:15.826351    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:14:44.989198 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:14:44.989158    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:14:53.963502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:14:53.963464    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:15:00.166931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.166896    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:00.170549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.170534    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:00.170667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.170650    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.173743 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.173728    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.223007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.222972    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:15:00.244522 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.244483    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjjm\" (UniqueName: \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.244658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.244626    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.244717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.244660    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.244717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.244682    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wgv\" (UniqueName: \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.276442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.276408    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:00.278340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.278314    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:00.345439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.345406    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wgv\" (UniqueName: \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.345617 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.345458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjjm\" (UniqueName: \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.345617 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.345551    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.345617 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.345587    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.346222 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.346205    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.346270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.346205    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.411325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.411293    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wgv\" (UniqueName: \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\") pod \"maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.411474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.411456    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjjm\" (UniqueName: \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\") pod \"maas-api-key-cleanup-29891595-mrz9z\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.484851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.484820    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:00.490790 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.490773    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:00.866608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.866584    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:00.868877 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:15:00.868846    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de7790e_2437_444b_bd9b_574bad27ae1f.slice/crio-03d123e32d0d6167dfb2c585206f7071614647794ad757843f2b3f73291de168 WatchSource:0}: Error finding container 03d123e32d0d6167dfb2c585206f7071614647794ad757843f2b3f73291de168: Status 404 returned error can't find the container with id 03d123e32d0d6167dfb2c585206f7071614647794ad757843f2b3f73291de168
Nov 01 01:15:00.916585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:00.916556    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:00.918356 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:15:00.918328    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b67956_f906_45f6_ac3a_692b36ddff19.slice/crio-2fefba023fe00beba0c3f7d5308fbef59e6de7e439974fbe9dfce24d47790eea WatchSource:0}: Error finding container 2fefba023fe00beba0c3f7d5308fbef59e6de7e439974fbe9dfce24d47790eea: Status 404 returned error can't find the container with id 2fefba023fe00beba0c3f7d5308fbef59e6de7e439974fbe9dfce24d47790eea
Nov 01 01:15:01.176984 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:01.176938    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerStarted","Data":"adf8e59a52af20931f6b075b0b5d3199ac0e1c1dc8705417a7c90f407e43cb22"}
Nov 01 01:15:01.177493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:01.177011    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerStarted","Data":"03d123e32d0d6167dfb2c585206f7071614647794ad757843f2b3f73291de168"}
Nov 01 01:15:01.178373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:01.178351    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" event={"ID":"f3b67956-f906-45f6-ac3a-692b36ddff19","Type":"ContainerStarted","Data":"2fefba023fe00beba0c3f7d5308fbef59e6de7e439974fbe9dfce24d47790eea"}
Nov 01 01:15:01.293522 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:01.293464    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" podStartSLOduration=1.293451001 podStartE2EDuration="1.293451001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:15:01.29221512 +0000 UTC m=+3090.969802068" watchObservedRunningTime="2026-11-01 01:15:01.293451001 +0000 UTC m=+3090.971037946"
Nov 01 01:15:02.184220 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.184185    4493 generic.go:356] "Generic (PLEG): container finished" podID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerID="7f2d547619370ac19ca832e1461d7bdfa5a4fa2b66202077992aa9e996823344" exitCode=60
Nov 01 01:15:02.184639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.184266    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" event={"ID":"f3b67956-f906-45f6-ac3a-692b36ddff19","Type":"ContainerDied","Data":"7f2d547619370ac19ca832e1461d7bdfa5a4fa2b66202077992aa9e996823344"}
Nov 01 01:15:02.184639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.184508    4493 scope.go:122] "RemoveContainer" containerID="7f2d547619370ac19ca832e1461d7bdfa5a4fa2b66202077992aa9e996823344"
Nov 01 01:15:02.186026 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.185989    4493 generic.go:356] "Generic (PLEG): container finished" podID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerID="adf8e59a52af20931f6b075b0b5d3199ac0e1c1dc8705417a7c90f407e43cb22" exitCode=60
Nov 01 01:15:02.186111 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.186032    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerDied","Data":"adf8e59a52af20931f6b075b0b5d3199ac0e1c1dc8705417a7c90f407e43cb22"}
Nov 01 01:15:02.186334 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:02.186320    4493 scope.go:122] "RemoveContainer" containerID="adf8e59a52af20931f6b075b0b5d3199ac0e1c1dc8705417a7c90f407e43cb22"
Nov 01 01:15:03.192524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.192491    4493 generic.go:356] "Generic (PLEG): container finished" podID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerID="d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b" exitCode=60
Nov 01 01:15:03.192936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.192577    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerDied","Data":"d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b"}
Nov 01 01:15:03.192936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.192644    4493 scope.go:122] "RemoveContainer" containerID="adf8e59a52af20931f6b075b0b5d3199ac0e1c1dc8705417a7c90f407e43cb22"
Nov 01 01:15:03.192936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.192913    4493 scope.go:122] "RemoveContainer" containerID="d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b"
Nov 01 01:15:03.193267 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:03.193245    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc_odh-ai-gateway-infra(4de7790e-2437-444b-bd9b-574bad27ae1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f"
Nov 01 01:15:03.194609 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.194588    4493 generic.go:356] "Generic (PLEG): container finished" podID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerID="85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50" exitCode=60
Nov 01 01:15:03.194728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.194618    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" event={"ID":"f3b67956-f906-45f6-ac3a-692b36ddff19","Type":"ContainerDied","Data":"85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50"}
Nov 01 01:15:03.194950 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.194935    4493 scope.go:122] "RemoveContainer" containerID="85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50"
Nov 01 01:15:03.195236 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:03.195200    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891595-mrz9z_odh-ai-gateway-infra(f3b67956-f906-45f6-ac3a-692b36ddff19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19"
Nov 01 01:15:03.204587 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:03.204554    4493 scope.go:122] "RemoveContainer" containerID="7f2d547619370ac19ca832e1461d7bdfa5a4fa2b66202077992aa9e996823344"
Nov 01 01:15:04.202212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:04.202181    4493 scope.go:122] "RemoveContainer" containerID="85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50"
Nov 01 01:15:04.202663 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:04.202497    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891595-mrz9z_odh-ai-gateway-infra(f3b67956-f906-45f6-ac3a-692b36ddff19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19"
Nov 01 01:15:04.203999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:04.203979    4493 scope.go:122] "RemoveContainer" containerID="d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b"
Nov 01 01:15:04.204214 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:04.204198    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc_odh-ai-gateway-infra(4de7790e-2437-444b-bd9b-574bad27ae1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f"
Nov 01 01:15:14.989065 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:14.988985    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:15:15.496344 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:15.496309    4493 scope.go:122] "RemoveContainer" containerID="d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b"
Nov 01 01:15:15.496534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:15.496418    4493 scope.go:122] "RemoveContainer" containerID="85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50"
Nov 01 01:15:16.266408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:16.266374    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerStarted","Data":"895604c01d362d5ee43e0cb984d45377f3be62394d177b05e66363b9a7f53c2b"}
Nov 01 01:15:16.268105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:16.268083    4493 generic.go:356] "Generic (PLEG): container finished" podID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerID="ec9bb972a45af76e27d5aeb9a99b135284415b170e717029bf02af859ff51ae9" exitCode=60
Nov 01 01:15:16.268270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:16.268115    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" event={"ID":"f3b67956-f906-45f6-ac3a-692b36ddff19","Type":"ContainerDied","Data":"ec9bb972a45af76e27d5aeb9a99b135284415b170e717029bf02af859ff51ae9"}
Nov 01 01:15:16.268270 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:16.268180    4493 scope.go:122] "RemoveContainer" containerID="85e676107d314087e829d2b39337ec131f9759f23e2b30d32350cf8125ceac50"
Nov 01 01:15:16.268499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:16.268483    4493 scope.go:122] "RemoveContainer" containerID="ec9bb972a45af76e27d5aeb9a99b135284415b170e717029bf02af859ff51ae9"
Nov 01 01:15:16.268772 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:16.268749    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891595-mrz9z_odh-ai-gateway-infra(f3b67956-f906-45f6-ac3a-692b36ddff19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19"
Nov 01 01:15:17.276073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.276041    4493 generic.go:356] "Generic (PLEG): container finished" podID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerID="895604c01d362d5ee43e0cb984d45377f3be62394d177b05e66363b9a7f53c2b" exitCode=60
Nov 01 01:15:17.276541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.276106    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerDied","Data":"895604c01d362d5ee43e0cb984d45377f3be62394d177b05e66363b9a7f53c2b"}
Nov 01 01:15:17.276541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.276181    4493 scope.go:122] "RemoveContainer" containerID="d2c8addd450e9e7e0b40bdd220458c19e1445ec60e650f165165b2165b3e539b"
Nov 01 01:15:17.276541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.276520    4493 scope.go:122] "RemoveContainer" containerID="895604c01d362d5ee43e0cb984d45377f3be62394d177b05e66363b9a7f53c2b"
Nov 01 01:15:17.276798 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:17.276776    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc_odh-ai-gateway-infra(4de7790e-2437-444b-bd9b-574bad27ae1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f"
Nov 01 01:15:17.392232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.392197    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:17.508396 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.508365    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:17.655434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.655401    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:17.728931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.728897    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\") pod \"f3b67956-f906-45f6-ac3a-692b36ddff19\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") "
Nov 01 01:15:17.729123 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.728982    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\" (UniqueName: \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\") pod \"f3b67956-f906-45f6-ac3a-692b36ddff19\" (UID: \"f3b67956-f906-45f6-ac3a-692b36ddff19\") "
Nov 01 01:15:17.729267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.729244    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca" pod "f3b67956-f906-45f6-ac3a-692b36ddff19" (UID: "f3b67956-f906-45f6-ac3a-692b36ddff19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:15:17.729405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.729337    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b67956-f906-45f6-ac3a-692b36ddff19-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:15:17.731218 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.731197    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm" pod "f3b67956-f906-45f6-ac3a-692b36ddff19" (UID: "f3b67956-f906-45f6-ac3a-692b36ddff19"). InnerVolumeSpecName "kube-api-access-vkjjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:15:17.830884 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:17.830805    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjjm\" (UniqueName: \"kubernetes.io/projected/f3b67956-f906-45f6-ac3a-692b36ddff19-kube-api-access-vkjjm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:15:18.282492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.282463    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"
Nov 01 01:15:18.282969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.282499    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z" event={"ID":"f3b67956-f906-45f6-ac3a-692b36ddff19","Type":"ContainerDied","Data":"2fefba023fe00beba0c3f7d5308fbef59e6de7e439974fbe9dfce24d47790eea"}
Nov 01 01:15:18.282969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.282560    4493 scope.go:122] "RemoveContainer" containerID="ec9bb972a45af76e27d5aeb9a99b135284415b170e717029bf02af859ff51ae9"
Nov 01 01:15:18.429776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.429752    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:18.540515 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.540418    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\") pod \"4de7790e-2437-444b-bd9b-574bad27ae1f\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") "
Nov 01 01:15:18.540515 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.540509    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\" (UniqueName: \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\") pod \"4de7790e-2437-444b-bd9b-574bad27ae1f\" (UID: \"4de7790e-2437-444b-bd9b-574bad27ae1f\") "
Nov 01 01:15:18.540835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.540811    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca" pod "4de7790e-2437-444b-bd9b-574bad27ae1f" (UID: "4de7790e-2437-444b-bd9b-574bad27ae1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:15:18.540956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.540941    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7790e-2437-444b-bd9b-574bad27ae1f-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:15:18.542936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.542917    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv" pod "4de7790e-2437-444b-bd9b-574bad27ae1f" (UID: "4de7790e-2437-444b-bd9b-574bad27ae1f"). InnerVolumeSpecName "kube-api-access-l9wgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:15:18.564244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.564214    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:18.567374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.567342    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891595-mrz9z"]
Nov 01 01:15:18.641971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:18.641939    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wgv\" (UniqueName: \"kubernetes.io/projected/4de7790e-2437-444b-bd9b-574bad27ae1f-kube-api-access-l9wgv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:15:19.290155 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:19.290109    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"
Nov 01 01:15:19.290672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:19.290156    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc" event={"ID":"4de7790e-2437-444b-bd9b-574bad27ae1f","Type":"ContainerDied","Data":"03d123e32d0d6167dfb2c585206f7071614647794ad757843f2b3f73291de168"}
Nov 01 01:15:19.290672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:19.290226    4493 scope.go:122] "RemoveContainer" containerID="895604c01d362d5ee43e0cb984d45377f3be62394d177b05e66363b9a7f53c2b"
Nov 01 01:15:19.455353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:19.455318    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:19.505201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:19.505172    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-48d289-29891595-nz4cc"]
Nov 01 01:15:20.501308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:20.501276    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" path="/var/lib/kubelet/pods/4de7790e-2437-444b-bd9b-574bad27ae1f/volumes"
Nov 01 01:15:20.501748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:20.501665    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" path="/var/lib/kubelet/pods/f3b67956-f906-45f6-ac3a-692b36ddff19/volumes"
Nov 01 01:15:29.349096 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:29.349060    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:15:31.642849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:15:31.642816    4493 scope.go:122] "RemoveContainer" containerID="a2e862f656f62ca12a0621e4a345d4ad2401d65e00ba5d052dc1a6b2f9614b09"
Nov 01 01:15:44.990594 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:15:44.990549    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:16:14.989326 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:16:14.989291    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:16:15.053990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:16:15.053955    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"]
Nov 01 01:16:15.103788 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:16:15.103747    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf7164-zngbj"]
Nov 01 01:16:16.501003 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:16:16.500972    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb" path="/var/lib/kubelet/pods/77f68edb-6caf-4aaa-96aa-d4fdee6dc9cb/volumes"
Nov 01 01:16:44.989099 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:16:44.989018    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:17:01.818819 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:01.818783    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:17:06.815004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:06.814967    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:17:14.989260 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:14.989212    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:17:16.838883 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:16.838840    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerName="main" containerID="cri-o://e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372" gracePeriod=45
Nov 01 01:17:16.850636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:16.850607    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": dial tcp 10.133.0.162:8000: connect: connection refused"
Nov 01 01:17:17.100193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.100108    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:17:17.178661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178625    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.178854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178721    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\" (UniqueName: \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.178854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178776    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\" (UniqueName: \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.178854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178808    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.179032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178904    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.179032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.178948    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\") pod \"36b2eafc-cbc7-4d73-a440-be20e3c19283\" (UID: \"36b2eafc-cbc7-4d73-a440-be20e3c19283\") "
Nov 01 01:17:17.179243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.179208    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:17.179243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.179219    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:17.179461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.179329    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:17.179461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.179407    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.179461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.179421    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.181250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.181223    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "kube-api-access-jr9pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:17.181475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.181450    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:17.181561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.181493    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs" pod "36b2eafc-cbc7-4d73-a440-be20e3c19283" (UID: "36b2eafc-cbc7-4d73-a440-be20e3c19283"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:17:17.279999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.279959    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.279999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.279992    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b2eafc-cbc7-4d73-a440-be20e3c19283-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.279999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.280000    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b2eafc-cbc7-4d73-a440-be20e3c19283-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.279999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.280009    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9pp\" (UniqueName: \"kubernetes.io/projected/36b2eafc-cbc7-4d73-a440-be20e3c19283-kube-api-access-jr9pp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:17.872856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.872820    4493 generic.go:356] "Generic (PLEG): container finished" podID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerID="e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372" exitCode=0
Nov 01 01:17:17.873369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.872872    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"
Nov 01 01:17:17.873369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.872907    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" event={"ID":"36b2eafc-cbc7-4d73-a440-be20e3c19283","Type":"ContainerDied","Data":"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"}
Nov 01 01:17:17.873369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.872968    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m" event={"ID":"36b2eafc-cbc7-4d73-a440-be20e3c19283","Type":"ContainerDied","Data":"a98c94677fbd018f9743ec2660479a495907de05377521db525984fdc15688b1"}
Nov 01 01:17:17.873369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.872992    4493 scope.go:122] "RemoveContainer" containerID="e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"
Nov 01 01:17:17.882192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.882114    4493 scope.go:122] "RemoveContainer" containerID="e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"
Nov 01 01:17:17.882435 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:17.882415    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372\": container with ID starting with e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372 not found: ID does not exist" containerID="e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"
Nov 01 01:17:17.882514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:17.882448    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372"} err="failed to get container status \"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372\": rpc error: code = NotFound desc = could not find container \"e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372\": container with ID starting with e913908d18d8382bdc5a4b606061172ffb783e6dec46f79beb3c786a9b239372 not found: ID does not exist"
Nov 01 01:17:18.038877 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:18.038844    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:17:18.088431 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:18.088394    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7fdf2c/test-model-7fdf2c-kserve-554dc78bf5-jrw8m"]
Nov 01 01:17:18.502418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:18.502386    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" path="/var/lib/kubelet/pods/36b2eafc-cbc7-4d73-a440-be20e3c19283/volumes"
Nov 01 01:17:22.684477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:22.684444    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:17:22.684975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:22.684690    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy" containerID="cri-o://5bd64dbddac6b655a7e641143bbab9345748cda318337c71ab5b9018b38240d9" gracePeriod=30
Nov 01 01:17:22.734699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:22.734655    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:17:22.734898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:22.734852    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy" containerID="cri-o://084e96efa151adfa70f1b43bac55e5474f40425ef567af0f21124f38d9b82117" gracePeriod=30
Nov 01 01:17:25.701368 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:25.701333    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:17:27.929620 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:27.929576    4493 generic.go:356] "Generic (PLEG): container finished" podID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerID="5bd64dbddac6b655a7e641143bbab9345748cda318337c71ab5b9018b38240d9" exitCode=0
Nov 01 01:17:27.930061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:27.929703    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" event={"ID":"de838895-1c79-4c6d-bab6-7aca01a66a04","Type":"ContainerDied","Data":"5bd64dbddac6b655a7e641143bbab9345748cda318337c71ab5b9018b38240d9"}
Nov 01 01:17:27.931656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:27.931631    4493 generic.go:356] "Generic (PLEG): container finished" podID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerID="084e96efa151adfa70f1b43bac55e5474f40425ef567af0f21124f38d9b82117" exitCode=0
Nov 01 01:17:27.931785 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:27.931703    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" event={"ID":"8b77912c-ade6-4469-a0d7-4b5c05e72226","Type":"ContainerDied","Data":"084e96efa151adfa70f1b43bac55e5474f40425ef567af0f21124f38d9b82117"}
Nov 01 01:17:27.990651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:27.990624    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:17:28.015907 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.015883    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:17:28.086421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086384    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086469    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086519    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086570    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086593    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086639    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086674    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086724    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.086852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086771    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\") pod \"de838895-1c79-4c6d-bab6-7aca01a66a04\" (UID: \"de838895-1c79-4c6d-bab6-7aca01a66a04\") "
Nov 01 01:17:28.087065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.086928    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.087065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087015    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:17:28.087414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087218    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de838895-1c79-4c6d-bab6-7aca01a66a04-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.087414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087241    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.087537 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087427    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.087645 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087616    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.087712 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.087649    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.088939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.088908    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "kube-api-access-hxhfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:28.089035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.088983    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.089035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.088997    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:28.089332 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.089312    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo" pod "de838895-1c79-4c6d-bab6-7aca01a66a04" (UID: "de838895-1c79-4c6d-bab6-7aca01a66a04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:17:28.188197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188123    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188228    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188257    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188294    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188362    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188412    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188510    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188559    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188603    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\") pod \"8b77912c-ade6-4469-a0d7-4b5c05e72226\" (UID: \"8b77912c-ade6-4469-a0d7-4b5c05e72226\") "
Nov 01 01:17:28.188840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.188840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188677    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:17:28.188840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188752    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188965    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.188988    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189008    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189023    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189036    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189051    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189065    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhfx\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-kube-api-access-hxhfx\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189081    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77912c-ade6-4469-a0d7-4b5c05e72226-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189095    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189109    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de838895-1c79-4c6d-bab6-7aca01a66a04-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.189624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189163    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.189624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.189233    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.190573 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.190542    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:28.190973 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.190948    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "kube-api-access-xmjxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:28.191313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.191290    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:28.191313 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.191301    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo" pod "8b77912c-ade6-4469-a0d7-4b5c05e72226" (UID: "8b77912c-ade6-4469-a0d7-4b5c05e72226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290072    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjxj\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-kube-api-access-xmjxj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290099    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290109    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290118    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290126    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.290168 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.290152    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77912c-ade6-4469-a0d7-4b5c05e72226-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:28.937233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.937199    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"
Nov 01 01:17:28.937233 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.937219    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7" event={"ID":"de838895-1c79-4c6d-bab6-7aca01a66a04","Type":"ContainerDied","Data":"57cc84f0120596362a1822d0a6fd38617c714d521d012fd7a3cb42f00b6fe3d2"}
Nov 01 01:17:28.937756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.937279    4493 scope.go:122] "RemoveContainer" containerID="5bd64dbddac6b655a7e641143bbab9345748cda318337c71ab5b9018b38240d9"
Nov 01 01:17:28.938660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.938640    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb" event={"ID":"8b77912c-ade6-4469-a0d7-4b5c05e72226","Type":"ContainerDied","Data":"95ee21442a1a66902207b5bb99bae3793623dcf95a4350b087de93631a46fb4c"}
Nov 01 01:17:28.938735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.938708    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"
Nov 01 01:17:28.946366 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:28.946347    4493 scope.go:122] "RemoveContainer" containerID="084e96efa151adfa70f1b43bac55e5474f40425ef567af0f21124f38d9b82117"
Nov 01 01:17:29.102460 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:29.102429    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:17:29.152225 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:29.152196    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2xrm7"]
Nov 01 01:17:29.225614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:29.225583    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:17:29.241162 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:29.241103    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p9ldb"]
Nov 01 01:17:30.501600 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:30.501569    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" path="/var/lib/kubelet/pods/8b77912c-ade6-4469-a0d7-4b5c05e72226/volumes"
Nov 01 01:17:30.502056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:30.502041    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" path="/var/lib/kubelet/pods/de838895-1c79-4c6d-bab6-7aca01a66a04/volumes"
Nov 01 01:17:31.677625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:31.677595    4493 scope.go:122] "RemoveContainer" containerID="27d046ae08c846d5b6692294b5f32b514075713a0d6fcfdfd427b47f806dac36"
Nov 01 01:17:31.686309 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:31.686287    4493 scope.go:122] "RemoveContainer" containerID="ea9449b6477c297afc206fd3b7011fa3d4bd45b36dc75ce0e65875ab68a97d29"
Nov 01 01:17:39.113205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:39.113171    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:17:39.113710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:39.113408    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" containerID="cri-o://b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8" gracePeriod=30
Nov 01 01:17:39.163033 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:39.162991    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:17:39.163323 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:39.163277    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy" containerID="cri-o://571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2" gracePeriod=30
Nov 01 01:17:42.460051 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:42.460017    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:17:42.770109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:42.770079    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:17:42.876507 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:42.876477    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:17:44.409231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.409199    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:17:44.459524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.459492    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.464428    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.464484    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.464529    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.464845    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465074    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465091    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465170    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465212    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465300    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465394    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\") pod \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\" (UID: \"c0fd31e3-1bd6-4bd5-9a66-909108206b20\") "
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465651    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465667    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465390    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.465911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.465611    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.467176 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.466020    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.467425 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.467406    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:44.468566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.468540    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:17:44.468566 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.468546    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "kube-api-access-bd2f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:44.468701 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.468585    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy" pod "c0fd31e3-1bd6-4bd5-9a66-909108206b20" (UID: "c0fd31e3-1bd6-4bd5-9a66-909108206b20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.567183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567072    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567159    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567231    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567264    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567362    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567467    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567523    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567542    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567584    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\") pod \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\" (UID: \"f6f6e54c-0449-4f99-bbe7-bf13739a9525\") "
Nov 01 01:17:44.567879 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567679    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.567939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567919    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.567996 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567968    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.567996 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567983    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.567997    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568014    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd2f8\" (UniqueName: \"kubernetes.io/projected/c0fd31e3-1bd6-4bd5-9a66-909108206b20-kube-api-access-bd2f8\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568029    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568044    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568057    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fd31e3-1bd6-4bd5-9a66-909108206b20-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.568627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568339    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.568627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568360    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.568751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568641    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:17:44.568751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.568720    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.570218 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.570195    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "kube-api-access-wbzvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:44.570320 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.570188    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:17:44.570464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.570446    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:17:44.570816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.570798    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token" pod "f6f6e54c-0449-4f99-bbe7-bf13739a9525" (UID: "f6f6e54c-0449-4f99-bbe7-bf13739a9525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669317    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669347    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669359    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669371    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669384    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669396    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669408    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6e54c-0449-4f99-bbe7-bf13739a9525-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.669726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:44.669421    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzvl\" (UniqueName: \"kubernetes.io/projected/f6f6e54c-0449-4f99-bbe7-bf13739a9525-kube-api-access-wbzvl\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:44.989265 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:44.989232    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:17:45.020262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.020232    4493 generic.go:356] "Generic (PLEG): container finished" podID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerID="571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2" exitCode=0
Nov 01 01:17:45.020487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.020272    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"
Nov 01 01:17:45.020487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.020311    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerDied","Data":"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"}
Nov 01 01:17:45.020487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.020366    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz" event={"ID":"f6f6e54c-0449-4f99-bbe7-bf13739a9525","Type":"ContainerDied","Data":"631ab8340219f37bb5829467a86c2e1da05cf13c70cb63f598be73ff8ce07987"}
Nov 01 01:17:45.020487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.020397    4493 scope.go:122] "RemoveContainer" containerID="571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"
Nov 01 01:17:45.021607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.021578    4493 generic.go:356] "Generic (PLEG): container finished" podID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerID="b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8" exitCode=0
Nov 01 01:17:45.021698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.021619    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" event={"ID":"c0fd31e3-1bd6-4bd5-9a66-909108206b20","Type":"ContainerDied","Data":"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"}
Nov 01 01:17:45.021698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.021638    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"
Nov 01 01:17:45.021698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.021667    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" event={"ID":"c0fd31e3-1bd6-4bd5-9a66-909108206b20","Type":"ContainerDied","Data":"b20866c070ec80fb2126ab8f50afafe0d6137f9c61ed81fd5ddbf7ff26c88942"}
Nov 01 01:17:45.030064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.030047    4493 scope.go:122] "RemoveContainer" containerID="d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"
Nov 01 01:17:45.038049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.038025    4493 scope.go:122] "RemoveContainer" containerID="571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"
Nov 01 01:17:45.038324 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:45.038301    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2\": container with ID starting with 571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2 not found: ID does not exist" containerID="571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"
Nov 01 01:17:45.038431 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.038333    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2"} err="failed to get container status \"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2\": rpc error: code = NotFound desc = could not find container \"571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2\": container with ID starting with 571852e93c2057770d01f28850ba769ccf5e9b8ec726dce307f2791128b4cbe2 not found: ID does not exist"
Nov 01 01:17:45.038431 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.038356    4493 scope.go:122] "RemoveContainer" containerID="d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"
Nov 01 01:17:45.038643 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:45.038626    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf\": container with ID starting with d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf not found: ID does not exist" containerID="d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"
Nov 01 01:17:45.038686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.038651    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf"} err="failed to get container status \"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf\": rpc error: code = NotFound desc = could not find container \"d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf\": container with ID starting with d0fd12682bd62474176197f4e67238a573126a48e8cd85443989f7742371ecaf not found: ID does not exist"
Nov 01 01:17:45.038686 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.038668    4493 scope.go:122] "RemoveContainer" containerID="b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"
Nov 01 01:17:45.046243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.046206    4493 scope.go:122] "RemoveContainer" containerID="b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"
Nov 01 01:17:45.046479 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:45.046462    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8\": container with ID starting with b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8 not found: ID does not exist" containerID="b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"
Nov 01 01:17:45.046533 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.046489    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8"} err="failed to get container status \"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8\": rpc error: code = NotFound desc = could not find container \"b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8\": container with ID starting with b5d970bbd5e15108f10da8e7c088b07914389e0b1f4c2a891ad6e5b70a6e1bd8 not found: ID does not exist"
Nov 01 01:17:45.187678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.187647    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:17:45.237182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.237119    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7"]
Nov 01 01:17:45.351249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.351168    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:17:45.357973 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.357946    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-thnrz"]
Nov 01 01:17:45.405639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.405605    4493 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:17:45.405805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:45.405676    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-xd2x7" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:17:46.501487 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:46.501456    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" path="/var/lib/kubelet/pods/c0fd31e3-1bd6-4bd5-9a66-909108206b20/volumes"
Nov 01 01:17:46.501905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:46.501891    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" path="/var/lib/kubelet/pods/f6f6e54c-0449-4f99-bbe7-bf13739a9525/volumes"
Nov 01 01:17:47.771036 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:47.770989    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing" containerID="cri-o://7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c" gracePeriod=25
Nov 01 01:17:47.877592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:47.877544    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing" containerID="cri-o://7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1" gracePeriod=25
Nov 01 01:17:48.245011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:48.244978    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:17:48.245252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:48.245229    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" podUID="7ebb6a93-3253-42a3-be62-fc124c12bb42" containerName="maas-api" containerID="cri-o://5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682" gracePeriod=30
Nov 01 01:17:49.489647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.489622    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:17:49.614340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.614302    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\") pod \"7ebb6a93-3253-42a3-be62-fc124c12bb42\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") "
Nov 01 01:17:49.614538 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.614396    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\" (UniqueName: \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\") pod \"7ebb6a93-3253-42a3-be62-fc124c12bb42\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") "
Nov 01 01:17:49.614951 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.614913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\") pod \"7ebb6a93-3253-42a3-be62-fc124c12bb42\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") "
Nov 01 01:17:49.615097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.614962    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\") pod \"7ebb6a93-3253-42a3-be62-fc124c12bb42\" (UID: \"7ebb6a93-3253-42a3-be62-fc124c12bb42\") "
Nov 01 01:17:49.615650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.615609    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca" pod "7ebb6a93-3253-42a3-be62-fc124c12bb42" (UID: "7ebb6a93-3253-42a3-be62-fc124c12bb42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:17:49.617356 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.617327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv" pod "7ebb6a93-3253-42a3-be62-fc124c12bb42" (UID: "7ebb6a93-3253-42a3-be62-fc124c12bb42"). InnerVolumeSpecName "kube-api-access-bf8vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:17:49.617822 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.617803    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls" pod "7ebb6a93-3253-42a3-be62-fc124c12bb42" (UID: "7ebb6a93-3253-42a3-be62-fc124c12bb42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:17:49.618258 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.618234    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs" pod "7ebb6a93-3253-42a3-be62-fc124c12bb42" (UID: "7ebb6a93-3253-42a3-be62-fc124c12bb42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:17:49.716652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.716620    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:49.716652 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.716653    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebb6a93-3253-42a3-be62-fc124c12bb42-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:49.716843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.716668    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebb6a93-3253-42a3-be62-fc124c12bb42-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:49.716843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:49.716682    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8vv\" (UniqueName: \"kubernetes.io/projected/7ebb6a93-3253-42a3-be62-fc124c12bb42-kube-api-access-bf8vv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:17:50.045808 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.045772    4493 generic.go:356] "Generic (PLEG): container finished" podID="7ebb6a93-3253-42a3-be62-fc124c12bb42" containerID="5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682" exitCode=0
Nov 01 01:17:50.046015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.045819    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"
Nov 01 01:17:50.046015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.045838    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" event={"ID":"7ebb6a93-3253-42a3-be62-fc124c12bb42","Type":"ContainerDied","Data":"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"}
Nov 01 01:17:50.046015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.045884    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw" event={"ID":"7ebb6a93-3253-42a3-be62-fc124c12bb42","Type":"ContainerDied","Data":"bdd4a2496cd5bf932f3a168a275e51de44a2b57319e8ec6cc04e3c7d207d6d56"}
Nov 01 01:17:50.046015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.045905    4493 scope.go:122] "RemoveContainer" containerID="5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"
Nov 01 01:17:50.056543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.056522    4493 scope.go:122] "RemoveContainer" containerID="5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"
Nov 01 01:17:50.056839 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:17:50.056819    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682\": container with ID starting with 5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682 not found: ID does not exist" containerID="5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"
Nov 01 01:17:50.056881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.056851    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682"} err="failed to get container status \"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682\": rpc error: code = NotFound desc = could not find container \"5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682\": container with ID starting with 5ed25dbe89958b7410dead1731cf79330322829ee32e058d7a2360dd924aa682 not found: ID does not exist"
Nov 01 01:17:50.214015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.213980    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:17:50.221374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.221337    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fdf2c-5b9cf9fd96-h4xkw"]
Nov 01 01:17:50.500930 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:50.500900    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebb6a93-3253-42a3-be62-fc124c12bb42" path="/var/lib/kubelet/pods/7ebb6a93-3253-42a3-be62-fc124c12bb42/volumes"
Nov 01 01:17:53.102451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:53.102406    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:17:53.102957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:53.102458    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:17:54.263992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:54.263955    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:17:58.611257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611226    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"]
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611617    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611633    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611647    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611652    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611659    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611663    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611667 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611671    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611676    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611683    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7ebb6a93-3253-42a3-be62-fc124c12bb42" containerName="maas-api"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611687    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebb6a93-3253-42a3-be62-fc124c12bb42" containerName="maas-api"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611698    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611706    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611723    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611729    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611736    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611740    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611748    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611753    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611763    4493 cpu_manager.go:414] "containerMap: removing container" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611767    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611779    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611787    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611797    4493 cpu_manager.go:414] "containerMap: removing container" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerName="main"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611802    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerName="main"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611873    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611885    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f6e54c-0449-4f99-bbe7-bf13739a9525" containerName="istio-proxy"
Nov 01 01:17:58.611885 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611892    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611898    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebb6a93-3253-42a3-be62-fc124c12bb42" containerName="maas-api"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611903    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611910    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fd31e3-1bd6-4bd5-9a66-909108206b20" containerName="istio-proxy"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611915    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611919    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de7790e-2437-444b-bd9b-574bad27ae1f" containerName="cleanup"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611926    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="de838895-1c79-4c6d-bab6-7aca01a66a04" containerName="istio-proxy"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611932    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b77912c-ade6-4469-a0d7-4b5c05e72226" containerName="istio-proxy"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611937    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b2eafc-cbc7-4d73-a440-be20e3c19283" containerName="main"
Nov 01 01:17:58.612485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.611942    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:17:58.615965 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.615948    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.668509 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.668478    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:17:58.697407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.697367    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfcg\" (UniqueName: \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.697571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.697447    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.697571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.697523    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.719061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.719031    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:17:58.719061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.719062    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:17:58.719338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.719085    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:17:58.720896 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.720873    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"]
Nov 01 01:17:58.798603 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.798570    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfcg\" (UniqueName: \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.798776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.798611    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.798776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.798650    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.799385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.799361    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.801141 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.801109    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.860881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.860846    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfcg\" (UniqueName: \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\") pod \"maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:58.928523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:58.928444    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:17:59.301854 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:17:59.301830    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"]
Nov 01 01:17:59.304342 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:17:59.304314    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe413fc5_0d1b_4324_839a_e97d85028650.slice/crio-ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d WatchSource:0}: Error finding container ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d: Status 404 returned error can't find the container with id ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d
Nov 01 01:18:00.093306 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:00.093270    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg" event={"ID":"fe413fc5-0d1b-4324-839a-e97d85028650","Type":"ContainerStarted","Data":"ac2e9a08cd6ce9c0854f91d8efaf979ebc18d34242a9742da3412d12c2f6da4c"}
Nov 01 01:18:00.093735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:00.093353    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg" event={"ID":"fe413fc5-0d1b-4324-839a-e97d85028650","Type":"ContainerStarted","Data":"ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d"}
Nov 01 01:18:00.208920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:00.208866    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg" podStartSLOduration=2.2088536149999998 podStartE2EDuration="2.208853615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:18:00.208632166 +0000 UTC m=+3269.886219110" watchObservedRunningTime="2026-11-01 01:18:00.208853615 +0000 UTC m=+3269.886440560"
Nov 01 01:18:01.100013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:01.099973    4493 generic.go:356] "Generic (PLEG): container finished" podID="fe413fc5-0d1b-4324-839a-e97d85028650" containerID="ac2e9a08cd6ce9c0854f91d8efaf979ebc18d34242a9742da3412d12c2f6da4c" exitCode=0
Nov 01 01:18:01.100435 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:01.100070    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg" event={"ID":"fe413fc5-0d1b-4324-839a-e97d85028650","Type":"ContainerDied","Data":"ac2e9a08cd6ce9c0854f91d8efaf979ebc18d34242a9742da3412d12c2f6da4c"}
Nov 01 01:18:02.243288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.243263    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:18:02.331868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.331831    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\") pod \"fe413fc5-0d1b-4324-839a-e97d85028650\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") "
Nov 01 01:18:02.332044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.331900    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\") pod \"fe413fc5-0d1b-4324-839a-e97d85028650\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") "
Nov 01 01:18:02.332044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.331940    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\" (UniqueName: \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\") pod \"fe413fc5-0d1b-4324-839a-e97d85028650\" (UID: \"fe413fc5-0d1b-4324-839a-e97d85028650\") "
Nov 01 01:18:02.332263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.332238    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca" pod "fe413fc5-0d1b-4324-839a-e97d85028650" (UID: "fe413fc5-0d1b-4324-839a-e97d85028650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:18:02.334216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.334188    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg" pod "fe413fc5-0d1b-4324-839a-e97d85028650" (UID: "fe413fc5-0d1b-4324-839a-e97d85028650"). InnerVolumeSpecName "kube-api-access-7dfcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:02.334216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.334198    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca" pod "fe413fc5-0d1b-4324-839a-e97d85028650" (UID: "fe413fc5-0d1b-4324-839a-e97d85028650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:18:02.433557 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.433467    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:02.433557 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.433501    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe413fc5-0d1b-4324-839a-e97d85028650-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:02.433557 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:02.433510    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfcg\" (UniqueName: \"kubernetes.io/projected/fe413fc5-0d1b-4324-839a-e97d85028650-kube-api-access-7dfcg\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:03.102183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:03.102127    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:18:03.102375 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:03.102152    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 01:18:03.110187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:03.110160    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"
Nov 01 01:18:03.110332 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:03.110183    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg" event={"ID":"fe413fc5-0d1b-4324-839a-e97d85028650","Type":"ContainerDied","Data":"ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d"}
Nov 01 01:18:03.110332 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:03.110225    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4890ae4aea55cb5b3417b7e94acd0693e1ff0a9b58bd00264e8bd4c8e48c2d"
Nov 01 01:18:08.018626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.018594    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:18:08.123657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.123633    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:18:08.134274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.134249    4493 generic.go:356] "Generic (PLEG): container finished" podID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerID="7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1" exitCode=0
Nov 01 01:18:08.134446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.134352    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" event={"ID":"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5","Type":"ContainerDied","Data":"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"}
Nov 01 01:18:08.134446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.134359    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"
Nov 01 01:18:08.134446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.134414    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl" event={"ID":"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5","Type":"ContainerDied","Data":"9c0e2c6fa87a99578627a6adfac69ad0ef8ffbac1d445166cf02a2bf6e8ea04d"}
Nov 01 01:18:08.134623 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.134452    4493 scope.go:122] "RemoveContainer" containerID="7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"
Nov 01 01:18:08.136402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.136378    4493 generic.go:356] "Generic (PLEG): container finished" podID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerID="7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c" exitCode=0
Nov 01 01:18:08.136519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.136459    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" event={"ID":"93aa87ef-2733-4911-9fc3-caf51fb5d669","Type":"ContainerDied","Data":"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"}
Nov 01 01:18:08.136519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.136496    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj" event={"ID":"93aa87ef-2733-4911-9fc3-caf51fb5d669","Type":"ContainerDied","Data":"a7237f408f50be56fdaf909c377976d1d9ad64a6ccf69f58ca1d0b8f064ce317"}
Nov 01 01:18:08.136624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.136557    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"
Nov 01 01:18:08.144803 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.144784    4493 scope.go:122] "RemoveContainer" containerID="7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"
Nov 01 01:18:08.145113 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:08.145095    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1\": container with ID starting with 7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1 not found: ID does not exist" containerID="7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"
Nov 01 01:18:08.145205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.145125    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1"} err="failed to get container status \"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1\": rpc error: code = NotFound desc = could not find container \"7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1\": container with ID starting with 7c4de9f75e67555119969e49c0f304fe64eccdc8bf9f436d6f98f2378d383ea1 not found: ID does not exist"
Nov 01 01:18:08.145205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.145166    4493 scope.go:122] "RemoveContainer" containerID="7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"
Nov 01 01:18:08.152943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.152926    4493 scope.go:122] "RemoveContainer" containerID="7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"
Nov 01 01:18:08.153201 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:08.153173    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c\": container with ID starting with 7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c not found: ID does not exist" containerID="7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"
Nov 01 01:18:08.153297 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.153209    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c"} err="failed to get container status \"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c\": rpc error: code = NotFound desc = could not find container \"7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c\": container with ID starting with 7f9267a0e5273578e4c95fc2a49a7ec1cf82ade46303b10adc60fa9d624b2d7c not found: ID does not exist"
Nov 01 01:18:08.185512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.185489    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\") pod \"93aa87ef-2733-4911-9fc3-caf51fb5d669\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") "
Nov 01 01:18:08.185644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.185629    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\" (UniqueName: \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\") pod \"93aa87ef-2733-4911-9fc3-caf51fb5d669\" (UID: \"93aa87ef-2733-4911-9fc3-caf51fb5d669\") "
Nov 01 01:18:08.185898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.185873    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume" pod "93aa87ef-2733-4911-9fc3-caf51fb5d669" (UID: "93aa87ef-2733-4911-9fc3-caf51fb5d669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:18:08.187943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.187923    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq" pod "93aa87ef-2733-4911-9fc3-caf51fb5d669" (UID: "93aa87ef-2733-4911-9fc3-caf51fb5d669"). InnerVolumeSpecName "kube-api-access-xvtrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:08.286895 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.286860    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\") pod \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") "
Nov 01 01:18:08.287097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.286943    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\" (UniqueName: \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\") pod \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\" (UID: \"c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5\") "
Nov 01 01:18:08.287207 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.287100    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvtrq\" (UniqueName: \"kubernetes.io/projected/93aa87ef-2733-4911-9fc3-caf51fb5d669-kube-api-access-xvtrq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:08.287207 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.287112    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93aa87ef-2733-4911-9fc3-caf51fb5d669-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:08.287372 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.287340    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume" pod "c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" (UID: "c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:18:08.289125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.289106    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd" pod "c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" (UID: "c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5"). InnerVolumeSpecName "kube-api-access-xxpsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:08.387941 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.387862    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpsd\" (UniqueName: \"kubernetes.io/projected/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-kube-api-access-xxpsd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:08.387941 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.387890    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:08.555526 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.555492    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:18:08.559142 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.559107    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fdf2c-767fc7b559-q22bl"]
Nov 01 01:18:08.669058 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.668979    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:18:08.673764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:08.673734    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fdf2c-fc47c4588-jsptj"]
Nov 01 01:18:10.501474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:10.501444    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" path="/var/lib/kubelet/pods/93aa87ef-2733-4911-9fc3-caf51fb5d669/volumes"
Nov 01 01:18:10.501937 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:10.501807    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" path="/var/lib/kubelet/pods/c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5/volumes"
Nov 01 01:18:14.989585 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:14.989515    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:18:16.857163 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:16.857116    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"]
Nov 01 01:18:16.907710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:16.907676    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-48d289-kr7vg"]
Nov 01 01:18:18.500314 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:18.500282    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe413fc5-0d1b-4324-839a-e97d85028650" path="/var/lib/kubelet/pods/fe413fc5-0d1b-4324-839a-e97d85028650/volumes"
Nov 01 01:18:20.283924 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:20.283890    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:18:25.157195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157153    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"]
Nov 01 01:18:25.157659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157610    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fe413fc5-0d1b-4324-839a-e97d85028650" containerName="revoke-keys"
Nov 01 01:18:25.157659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157624    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe413fc5-0d1b-4324-839a-e97d85028650" containerName="revoke-keys"
Nov 01 01:18:25.157659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157642    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:18:25.157659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157647    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:18:25.157659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157660    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157665    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157674    4493 cpu_manager.go:414] "containerMap: removing container" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157679    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157730    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aa87ef-2733-4911-9fc3-caf51fb5d669" containerName="payload-processing"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157741    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe413fc5-0d1b-4324-839a-e97d85028650" containerName="revoke-keys"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157749    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b67956-f906-45f6-ac3a-692b36ddff19" containerName="cleanup"
Nov 01 01:18:25.157839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.157755    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c1b70d-4bc2-40fd-9deb-1cf61c44f5b5" containerName="payload-pre-processing"
Nov 01 01:18:25.162170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.162154    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.214715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.214681    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:18:25.246094 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.246061    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.246278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.246120    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.246340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.246304    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxlpv\" (UniqueName: \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.264308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.264281    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:18:25.264445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.264386    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:18:25.264445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.264391    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:18:25.265925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.265907    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"]
Nov 01 01:18:25.347050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.347019    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.347271 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.347102    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxlpv\" (UniqueName: \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.347271 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.347205    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.347769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.347748    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.349695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.349675    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.408829 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.408763    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxlpv\" (UniqueName: \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\") pod \"maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.474804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.474774    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:25.849408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:25.849380    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"]
Nov 01 01:18:25.852234 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:18:25.852198    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2a9868_acaf_4343_8116_de7696e2394d.slice/crio-60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248 WatchSource:0}: Error finding container 60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248: Status 404 returned error can't find the container with id 60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248
Nov 01 01:18:26.220285 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:26.220245    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x" event={"ID":"fb2a9868-acaf-4343-8116-de7696e2394d","Type":"ContainerStarted","Data":"354d9c9f83e4cd33f34edbfce563c17ce9596f785591d2209e3815f15b23d2c8"}
Nov 01 01:18:26.220708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:26.220314    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x" event={"ID":"fb2a9868-acaf-4343-8116-de7696e2394d","Type":"ContainerStarted","Data":"60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248"}
Nov 01 01:18:26.341717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:26.341667    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x" podStartSLOduration=1.341651253 podStartE2EDuration="1.341651253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:18:26.339573129 +0000 UTC m=+3296.017160074" watchObservedRunningTime="2026-11-01 01:18:26.341651253 +0000 UTC m=+3296.019238198"
Nov 01 01:18:27.226459 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:27.226422    4493 generic.go:356] "Generic (PLEG): container finished" podID="fb2a9868-acaf-4343-8116-de7696e2394d" containerID="354d9c9f83e4cd33f34edbfce563c17ce9596f785591d2209e3815f15b23d2c8" exitCode=0
Nov 01 01:18:27.226852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:27.226510    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x" event={"ID":"fb2a9868-acaf-4343-8116-de7696e2394d","Type":"ContainerDied","Data":"354d9c9f83e4cd33f34edbfce563c17ce9596f785591d2209e3815f15b23d2c8"}
Nov 01 01:18:28.364660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.364636    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:28.474656 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.474626    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\") pod \"fb2a9868-acaf-4343-8116-de7696e2394d\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") "
Nov 01 01:18:28.474859 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.474728    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\" (UniqueName: \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\") pod \"fb2a9868-acaf-4343-8116-de7696e2394d\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") "
Nov 01 01:18:28.474859 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.474769    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\") pod \"fb2a9868-acaf-4343-8116-de7696e2394d\" (UID: \"fb2a9868-acaf-4343-8116-de7696e2394d\") "
Nov 01 01:18:28.475052 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.475027    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca" pod "fb2a9868-acaf-4343-8116-de7696e2394d" (UID: "fb2a9868-acaf-4343-8116-de7696e2394d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:18:28.477083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.477053    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca" pod "fb2a9868-acaf-4343-8116-de7696e2394d" (UID: "fb2a9868-acaf-4343-8116-de7696e2394d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:18:28.477236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.477165    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv" pod "fb2a9868-acaf-4343-8116-de7696e2394d" (UID: "fb2a9868-acaf-4343-8116-de7696e2394d"). InnerVolumeSpecName "kube-api-access-hxlpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:28.575777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.575744    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxlpv\" (UniqueName: \"kubernetes.io/projected/fb2a9868-acaf-4343-8116-de7696e2394d-kube-api-access-hxlpv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:28.575777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.575771    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:28.575777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:28.575781    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a9868-acaf-4343-8116-de7696e2394d-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:29.236365 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:29.236338    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"
Nov 01 01:18:29.236365 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:29.236346    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x" event={"ID":"fb2a9868-acaf-4343-8116-de7696e2394d","Type":"ContainerDied","Data":"60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248"}
Nov 01 01:18:29.236599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:29.236389    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e9fd58dca370c399a9d30d306ea4e6bddf9b09d695cbf167dc6c17729b9248"
Nov 01 01:18:29.626987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:29.626905    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 01:18:29.645234 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:29.645198    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 01:18:36.388210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.388175    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"]
Nov 01 01:18:36.388598 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.388544    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb2a9868-acaf-4343-8116-de7696e2394d" containerName="revoke-keys"
Nov 01 01:18:36.388598 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.388555    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2a9868-acaf-4343-8116-de7696e2394d" containerName="revoke-keys"
Nov 01 01:18:36.388676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.388620    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2a9868-acaf-4343-8116-de7696e2394d" containerName="revoke-keys"
Nov 01 01:18:36.391625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.391609    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.448498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.448465    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.448678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.448507    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.448678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.448530    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knl6x\" (UniqueName: \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.496523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.496494    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:18:36.496733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.496541    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:18:36.496733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.496549    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:18:36.496733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.496604    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:18:36.500794 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.500774    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"]
Nov 01 01:18:36.549902 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.549870    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.550095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.549914    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.550095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.549937    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knl6x\" (UniqueName: \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.550597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.550570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.552484 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.552459    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.611666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.611634    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knl6x\" (UniqueName: \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\") pod \"maas-api-revoke-sub-8c93b493-d8crr\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:36.704472 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:36.704392    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:37.075746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:37.075723    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"]
Nov 01 01:18:37.077766 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:18:37.077737    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fe6766_0d13_4a66_920f_366546492797.slice/crio-deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68 WatchSource:0}: Error finding container deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68: Status 404 returned error can't find the container with id deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68
Nov 01 01:18:37.272822 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:37.272783    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr" event={"ID":"a5fe6766-0d13-4a66-920f-366546492797","Type":"ContainerStarted","Data":"deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68"}
Nov 01 01:18:38.280955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:38.280858    4493 generic.go:356] "Generic (PLEG): container finished" podID="a5fe6766-0d13-4a66-920f-366546492797" containerID="ad4c75fd99514a128230794f51075c8026bbca7c1dbc2052486761b16f6f2912" exitCode=0
Nov 01 01:18:38.280955 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:38.280908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr" event={"ID":"a5fe6766-0d13-4a66-920f-366546492797","Type":"ContainerDied","Data":"ad4c75fd99514a128230794f51075c8026bbca7c1dbc2052486761b16f6f2912"}
Nov 01 01:18:39.450074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.450052    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:39.577119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.577039    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\") pod \"a5fe6766-0d13-4a66-920f-366546492797\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") "
Nov 01 01:18:39.577119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.577112    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\" (UniqueName: \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\") pod \"a5fe6766-0d13-4a66-920f-366546492797\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") "
Nov 01 01:18:39.577376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.577251    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\") pod \"a5fe6766-0d13-4a66-920f-366546492797\" (UID: \"a5fe6766-0d13-4a66-920f-366546492797\") "
Nov 01 01:18:39.577611 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.577582    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca" pod "a5fe6766-0d13-4a66-920f-366546492797" (UID: "a5fe6766-0d13-4a66-920f-366546492797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:18:39.579462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.579440    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x" pod "a5fe6766-0d13-4a66-920f-366546492797" (UID: "a5fe6766-0d13-4a66-920f-366546492797"). InnerVolumeSpecName "kube-api-access-knl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:39.579762 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.579737    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca" pod "a5fe6766-0d13-4a66-920f-366546492797" (UID: "a5fe6766-0d13-4a66-920f-366546492797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:18:39.678905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.678871    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fe6766-0d13-4a66-920f-366546492797-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:39.678905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.678900    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fe6766-0d13-4a66-920f-366546492797-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:39.678905 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:39.678910    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knl6x\" (UniqueName: \"kubernetes.io/projected/a5fe6766-0d13-4a66-920f-366546492797-kube-api-access-knl6x\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:40.291822 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:40.291783    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr" event={"ID":"a5fe6766-0d13-4a66-920f-366546492797","Type":"ContainerDied","Data":"deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68"}
Nov 01 01:18:40.291822 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:40.291806    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"
Nov 01 01:18:40.292035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:40.291843    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deba8aed31fdc2c3ea6bdc00e44427d71149865217256c258360b5e9cb8a2b68"
Nov 01 01:18:44.648017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:44.647966    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main" containerID="cri-o://89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992" gracePeriod=45
Nov 01 01:18:44.669614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:44.669576    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main" containerID="cri-o://a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863" gracePeriod=45
Nov 01 01:18:44.911170 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:44.911084    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 01:18:44.939007 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:44.938984    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 01:18:44.989522 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:44.989485    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:18:45.036835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036797    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\" (UniqueName: \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036846    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036891    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036940    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036967    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.036993    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037029    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037065    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\" (UniqueName: \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037113    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037184    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037198    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037226    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\") pod \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\" (UID: \"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037260    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\") pod \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\" (UID: \"2e0bc473-76d4-4b7e-bec4-32930c9f8656\") "
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037261    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.037348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037313    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037521    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037575    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037667    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037688    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037718    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.037804 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037733    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.038147 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.037836    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.039847 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.039813    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.040000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.039878    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "kube-api-access-fj2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:45.040000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.039901    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "kube-api-access-vmm2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:18:45.040000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.039944    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs" pod "2e0bc473-76d4-4b7e-bec4-32930c9f8656" (UID: "2e0bc473-76d4-4b7e-bec4-32930c9f8656"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:18:45.040000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.039990    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:18:45.040171 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.040080    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm" pod "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" (UID: "cfb68e8e-7b6e-42b3-adfd-32b0d7694c85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:18:45.139203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139165    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139196    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139210    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2qj\" (UniqueName: \"kubernetes.io/projected/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-kube-api-access-fj2qj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139223    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139235    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0bc473-76d4-4b7e-bec4-32930c9f8656-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139246    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139257    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmm2t\" (UniqueName: \"kubernetes.io/projected/2e0bc473-76d4-4b7e-bec4-32930c9f8656-kube-api-access-vmm2t\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.139462 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.139269    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0bc473-76d4-4b7e-bec4-32930c9f8656-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:18:45.317041 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.317012    4493 generic.go:356] "Generic (PLEG): container finished" podID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerID="89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992" exitCode=0
Nov 01 01:18:45.317255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.317059    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"
Nov 01 01:18:45.317255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.317110    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" event={"ID":"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85","Type":"ContainerDied","Data":"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"}
Nov 01 01:18:45.317255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.317171    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h" event={"ID":"cfb68e8e-7b6e-42b3-adfd-32b0d7694c85","Type":"ContainerDied","Data":"343ae1dca90f0a8af4cffc7a994b2cd0b7f516ca222757c9825f121be3b96aad"}
Nov 01 01:18:45.317255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.317190    4493 scope.go:122] "RemoveContainer" containerID="89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"
Nov 01 01:18:45.318606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.318579    4493 generic.go:356] "Generic (PLEG): container finished" podID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerID="a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863" exitCode=0
Nov 01 01:18:45.318731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.318630    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"
Nov 01 01:18:45.318731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.318643    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" event={"ID":"2e0bc473-76d4-4b7e-bec4-32930c9f8656","Type":"ContainerDied","Data":"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"}
Nov 01 01:18:45.318731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.318675    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr" event={"ID":"2e0bc473-76d4-4b7e-bec4-32930c9f8656","Type":"ContainerDied","Data":"0ae73dc612351b19f9a3b1d00a932fcd90844df177952d09be7761b86803bbf0"}
Nov 01 01:18:45.325866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.325845    4493 scope.go:122] "RemoveContainer" containerID="89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"
Nov 01 01:18:45.326186 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:45.326162    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992\": container with ID starting with 89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992 not found: ID does not exist" containerID="89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"
Nov 01 01:18:45.326299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.326198    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992"} err="failed to get container status \"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992\": rpc error: code = NotFound desc = could not find container \"89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992\": container with ID starting with 89261aaedb17aaad2b12046f03c169a2c3022ff21ebf7a80ad31f5d128551992 not found: ID does not exist"
Nov 01 01:18:45.326299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.326224    4493 scope.go:122] "RemoveContainer" containerID="a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"
Nov 01 01:18:45.334256 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.334182    4493 scope.go:122] "RemoveContainer" containerID="a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"
Nov 01 01:18:45.334478 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:18:45.334461    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863\": container with ID starting with a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863 not found: ID does not exist" containerID="a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"
Nov 01 01:18:45.334543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.334484    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863"} err="failed to get container status \"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863\": rpc error: code = NotFound desc = could not find container \"a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863\": container with ID starting with a2ad51e5a4131e0ee67b43f75244a6cebaee91187f83cd3c9ac08e5b86b28863 not found: ID does not exist"
Nov 01 01:18:45.484363 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.484328    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 01:18:45.486970 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.486948    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/simulator-w4-43a0cf-kserve-5c9fbbc79b-cczlr"]
Nov 01 01:18:45.598666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.598583    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 01:18:45.604517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:45.604491    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-w4-43a0cf-kserve-699b64d9dd-ng29h"]
Nov 01 01:18:46.500650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:46.500619    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" path="/var/lib/kubelet/pods/2e0bc473-76d4-4b7e-bec4-32930c9f8656/volumes"
Nov 01 01:18:46.501039 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:46.501001    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" path="/var/lib/kubelet/pods/cfb68e8e-7b6e-42b3-adfd-32b0d7694c85/volumes"
Nov 01 01:18:53.205243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:53.205208    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:18:53.589488 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:53.589457    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"]
Nov 01 01:18:53.638756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:53.638723    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43a0cf-blb6x"]
Nov 01 01:18:54.501875 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:18:54.501833    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2a9868-acaf-4343-8116-de7696e2394d" path="/var/lib/kubelet/pods/fb2a9868-acaf-4343-8116-de7696e2394d/volumes"
Nov 01 01:19:08.268117 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:08.268082    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:19:14.989121 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:19:14.989079    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:19:23.655486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:23.655453    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:19:23.655998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:23.655659    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" containerID="cri-o://a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a" gracePeriod=30
Nov 01 01:19:28.565308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:28.565269    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t48rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:19:28.565771 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:28.565331    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:19:28.900486 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:28.900462    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:19:29.037196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037162    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037217    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037239    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037275    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037297    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037318    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037382    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037470 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037424    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\") pod \"4adf1d7e-1093-4ab5-9406-88d379726d9f\" (UID: \"4adf1d7e-1093-4ab5-9406-88d379726d9f\") "
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037478    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037599    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037618    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037685    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037777    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adf1d7e-1093-4ab5-9406-88d379726d9f-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.037805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037792    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.038373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037812    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.038373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037920    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:19:29.038373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.037830    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.039802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.039778    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:19:29.039931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.039821    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:19:29.039931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.039872    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "kube-api-access-r2h6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:19:29.039931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.039895    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token" pod "4adf1d7e-1093-4ab5-9406-88d379726d9f" (UID: "4adf1d7e-1093-4ab5-9406-88d379726d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:19:29.138923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.138848    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.138923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.138876    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.138923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.138885    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.138923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.138894    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2h6k\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-kube-api-access-r2h6k\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.138923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.138903    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adf1d7e-1093-4ab5-9406-88d379726d9f-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:29.511157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.511099    4493 generic.go:356] "Generic (PLEG): container finished" podID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerID="a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a" exitCode=0
Nov 01 01:19:29.511340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.511173    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"
Nov 01 01:19:29.511340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.511180    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" event={"ID":"4adf1d7e-1093-4ab5-9406-88d379726d9f","Type":"ContainerDied","Data":"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"}
Nov 01 01:19:29.511340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.511237    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm" event={"ID":"4adf1d7e-1093-4ab5-9406-88d379726d9f","Type":"ContainerDied","Data":"49808eaa139744295fdd73d88df37d3bfc3d9fca7215b62f71542f5f2ad90d03"}
Nov 01 01:19:29.511340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.511261    4493 scope.go:122] "RemoveContainer" containerID="a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"
Nov 01 01:19:29.520610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.520592    4493 scope.go:122] "RemoveContainer" containerID="a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"
Nov 01 01:19:29.520871 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:19:29.520851    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a\": container with ID starting with a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a not found: ID does not exist" containerID="a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"
Nov 01 01:19:29.520922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.520880    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a"} err="failed to get container status \"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a\": rpc error: code = NotFound desc = could not find container \"a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a\": container with ID starting with a79d4998513f75498f5fc54b05edc8280ca3e6a8f548621298ce065cf4b8555a not found: ID does not exist"
Nov 01 01:19:29.676987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.676944    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:19:29.726582 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:29.726553    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t48rm"]
Nov 01 01:19:30.500845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:30.500811    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" path="/var/lib/kubelet/pods/4adf1d7e-1093-4ab5-9406-88d379726d9f/volumes"
Nov 01 01:19:31.355068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355035    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"]
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355407    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355419    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355432    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355437    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355444    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a5fe6766-0d13-4a66-920f-366546492797" containerName="revoke-keys"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355450    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fe6766-0d13-4a66-920f-366546492797" containerName="revoke-keys"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355458    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main"
Nov 01 01:19:31.355461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355463    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main"
Nov 01 01:19:31.355777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355513    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb68e8e-7b6e-42b3-adfd-32b0d7694c85" containerName="main"
Nov 01 01:19:31.355777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355523    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adf1d7e-1093-4ab5-9406-88d379726d9f" containerName="istio-proxy"
Nov 01 01:19:31.355777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355529    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fe6766-0d13-4a66-920f-366546492797" containerName="revoke-keys"
Nov 01 01:19:31.355777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.355535    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0bc473-76d4-4b7e-bec4-32930c9f8656" containerName="main"
Nov 01 01:19:31.359440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.359411    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.411894 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.411859    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:19:31.460899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.460852    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.460899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.460903    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.461174 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.460928    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwm8\" (UniqueName: \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.461803 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.461783    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:19:31.461878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.461810    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:19:31.461878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.461811    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:19:31.463367 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.463342    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"]
Nov 01 01:19:31.561577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.561532    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.561577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.561585    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.561846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.561620    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwm8\" (UniqueName: \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.562301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.562277    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.564149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.564113    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.638543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.638471    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwm8\" (UniqueName: \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\") pod \"maas-api-revoke-sub-85237e45-tgvpv\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:31.672752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:31.672720    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:32.048979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:32.048947    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"]
Nov 01 01:19:32.052399 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:19:32.052371    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd0b394_0224_449f_8dfb_c8c73b31f8c7.slice/crio-6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e WatchSource:0}: Error finding container 6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e: Status 404 returned error can't find the container with id 6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e
Nov 01 01:19:32.525759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:32.525717    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv" event={"ID":"4cd0b394-0224-449f-8dfb-c8c73b31f8c7","Type":"ContainerStarted","Data":"564da360e983458886460a4c0cf4be1a146db9ea55addc2ff8bd6c6c3966f163"}
Nov 01 01:19:32.526193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:32.525780    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv" event={"ID":"4cd0b394-0224-449f-8dfb-c8c73b31f8c7","Type":"ContainerStarted","Data":"6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e"}
Nov 01 01:19:32.643184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:32.643103    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv" podStartSLOduration=1.643084114 podStartE2EDuration="1.643084114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:19:32.642823794 +0000 UTC m=+3362.320410739" watchObservedRunningTime="2026-11-01 01:19:32.643084114 +0000 UTC m=+3362.320671059"
Nov 01 01:19:33.531421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:33.531387    4493 generic.go:356] "Generic (PLEG): container finished" podID="4cd0b394-0224-449f-8dfb-c8c73b31f8c7" containerID="564da360e983458886460a4c0cf4be1a146db9ea55addc2ff8bd6c6c3966f163" exitCode=0
Nov 01 01:19:33.531809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:33.531448    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv" event={"ID":"4cd0b394-0224-449f-8dfb-c8c73b31f8c7","Type":"ContainerDied","Data":"564da360e983458886460a4c0cf4be1a146db9ea55addc2ff8bd6c6c3966f163"}
Nov 01 01:19:34.670082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.670056    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:34.789000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.788912    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\") pod \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") "
Nov 01 01:19:34.789000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.788962    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\" (UniqueName: \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\") pod \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") "
Nov 01 01:19:34.789249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.789032    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\") pod \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\" (UID: \"4cd0b394-0224-449f-8dfb-c8c73b31f8c7\") "
Nov 01 01:19:34.789651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.789611    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca" pod "4cd0b394-0224-449f-8dfb-c8c73b31f8c7" (UID: "4cd0b394-0224-449f-8dfb-c8c73b31f8c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:19:34.791802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.791779    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca" pod "4cd0b394-0224-449f-8dfb-c8c73b31f8c7" (UID: "4cd0b394-0224-449f-8dfb-c8c73b31f8c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:19:34.791928 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.791908    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8" pod "4cd0b394-0224-449f-8dfb-c8c73b31f8c7" (UID: "4cd0b394-0224-449f-8dfb-c8c73b31f8c7"). InnerVolumeSpecName "kube-api-access-8hwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:19:34.890550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.890500    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:34.890550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.890548    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:34.890550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:34.890558    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwm8\" (UniqueName: \"kubernetes.io/projected/4cd0b394-0224-449f-8dfb-c8c73b31f8c7-kube-api-access-8hwm8\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:19:35.540843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:35.540815    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"
Nov 01 01:19:35.541018 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:35.540809    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv" event={"ID":"4cd0b394-0224-449f-8dfb-c8c73b31f8c7","Type":"ContainerDied","Data":"6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e"}
Nov 01 01:19:35.541018 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:35.540935    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6851cb63a547d9b6c1712025d17a2efb72250c2fe6382bd758cc6d0637800a1e"
Nov 01 01:19:35.652402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:35.652370    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:19:38.817061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.817025    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"]
Nov 01 01:19:38.817485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.817384    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4cd0b394-0224-449f-8dfb-c8c73b31f8c7" containerName="revoke-keys"
Nov 01 01:19:38.817485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.817396    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd0b394-0224-449f-8dfb-c8c73b31f8c7" containerName="revoke-keys"
Nov 01 01:19:38.817485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.817467    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd0b394-0224-449f-8dfb-c8c73b31f8c7" containerName="revoke-keys"
Nov 01 01:19:38.820490 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.820475    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.873519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.873487    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\""
Nov 01 01:19:38.925074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925033    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925159    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925221    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925357    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925473 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925427    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925517    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925587 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925563    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925587    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"]
Nov 01 01:19:38.925641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925612    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:38.925747 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:38.925645    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7pf\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-kube-api-access-kl7pf\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.026791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026757    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.026791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026792    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026818    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026847    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026871    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7pf\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-kube-api-access-kl7pf\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026896    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026933    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.026964    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.027002    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.027300    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.027471    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027598 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.027540    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.027598 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.027542    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.028157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.028105    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.029541 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.029523    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.029684 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.029665    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.088185 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.088101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.088319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.088262    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7pf\" (UniqueName: \"kubernetes.io/projected/bc06b95d-b4c0-4dd9-822e-b0df3090ee29-kube-api-access-kl7pf\") pod \"maas-default-gateway-openshift-default-67bf986f99-48lrj\" (UID: \"bc06b95d-b4c0-4dd9-822e-b0df3090ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.133276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.133246    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:39.559735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.559602    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"]
Nov 01 01:19:39.562335 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:19:39.562307    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc06b95d_b4c0_4dd9_822e_b0df3090ee29.slice/crio-62da1467a267c9061a7b5c967e97991e1441d5a0e96b80f52f7db09a1cdcdeec WatchSource:0}: Error finding container 62da1467a267c9061a7b5c967e97991e1441d5a0e96b80f52f7db09a1cdcdeec: Status 404 returned error can't find the container with id 62da1467a267c9061a7b5c967e97991e1441d5a0e96b80f52f7db09a1cdcdeec
Nov 01 01:19:39.564391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.564359    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:19:39.564455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.564424    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:19:39.564493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:39.564456    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:19:40.562027 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:40.561985    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" event={"ID":"bc06b95d-b4c0-4dd9-822e-b0df3090ee29","Type":"ContainerStarted","Data":"6f73bf030764f253a544d239503156a540e43d880f1620f22fc66d6dbfe52792"}
Nov 01 01:19:40.562464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:40.562040    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" event={"ID":"bc06b95d-b4c0-4dd9-822e-b0df3090ee29","Type":"ContainerStarted","Data":"62da1467a267c9061a7b5c967e97991e1441d5a0e96b80f52f7db09a1cdcdeec"}
Nov 01 01:19:40.741159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:40.732603    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podStartSLOduration=2.73258334 podStartE2EDuration="2.73258334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:19:40.732330274 +0000 UTC m=+3370.409917219" watchObservedRunningTime="2026-11-01 01:19:40.73258334 +0000 UTC m=+3370.410170286"
Nov 01 01:19:41.136494 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:41.136461    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:41.141775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:41.141740    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:41.141958 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:41.141804    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:42.136692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:42.136597    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:42.137460 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:42.137424    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:43.134621 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:43.134583    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:43.134931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:43.134899    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:44.137607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:44.137563    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:44.138211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:44.138177    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:44.989261 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:19:44.989223    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:19:45.134592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:45.134553    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:45.134849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:45.134821    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:46.134562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:46.134517    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-48lrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 01:19:46.134969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:46.134583    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj" podUID="bc06b95d-b4c0-4dd9-822e-b0df3090ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 01:19:47.750257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:47.750221    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:47.750957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:47.750934    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:19:47.751719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:19:47.751692    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-48lrj"
Nov 01 01:20:14.989422 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:20:14.989385    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:20:32.836349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:32.836313    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:20:41.386208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.386174    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"]
Nov 01 01:20:41.390038 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.390010    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.442569 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.442538    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:20:41.445931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.445897    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.446090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.446058    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.446090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.446086    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mkd\" (UniqueName: \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.492360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.492329    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:20:41.492360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.492349    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:20:41.492360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.492331    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:20:41.493998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.493979    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"]
Nov 01 01:20:41.546750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.546717    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.546750 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.546753    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mkd\" (UniqueName: \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.546956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.546784    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.547461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.547435    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.549277 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.549257    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.608437 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.608402    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mkd\" (UniqueName: \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\") pod \"maas-api-revoke-sub-c1e271a9-4tgg5\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:41.703104 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:41.703028    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:42.079760 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:42.079617    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"]
Nov 01 01:20:42.082114 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:20:42.082086    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9c0ca_0f66_4529_8fd8_62f1a1f4a2f6.slice/crio-a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30 WatchSource:0}: Error finding container a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30: Status 404 returned error can't find the container with id a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30
Nov 01 01:20:42.839619 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:42.839583    4493 generic.go:356] "Generic (PLEG): container finished" podID="7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" containerID="cdc4feacb467f8f5942cbd75a939100ec952192a8d262904dde861acb50e5c8c" exitCode=0
Nov 01 01:20:42.840046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:42.839666    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5" event={"ID":"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6","Type":"ContainerDied","Data":"cdc4feacb467f8f5942cbd75a939100ec952192a8d262904dde861acb50e5c8c"}
Nov 01 01:20:42.840046 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:42.839720    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5" event={"ID":"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6","Type":"ContainerStarted","Data":"a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30"}
Nov 01 01:20:43.973417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:43.973392    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:44.067957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.067922    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\" (UniqueName: \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\") pod \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") "
Nov 01 01:20:44.068212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.067999    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\") pod \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") "
Nov 01 01:20:44.068212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.068027    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\") pod \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\" (UID: \"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6\") "
Nov 01 01:20:44.068406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.068382    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca" pod "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" (UID: "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:20:44.070374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.070341    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd" pod "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" (UID: "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6"). InnerVolumeSpecName "kube-api-access-59mkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:20:44.070482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.070394    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca" pod "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" (UID: "7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:20:44.169122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.169041    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:20:44.169122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.169072    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:20:44.169122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.169082    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59mkd\" (UniqueName: \"kubernetes.io/projected/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6-kube-api-access-59mkd\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:20:44.849838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.849751    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5" event={"ID":"7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6","Type":"ContainerDied","Data":"a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30"}
Nov 01 01:20:44.849838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.849774    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"
Nov 01 01:20:44.849838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:44.849799    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a450cc3d9087526fedb6a7707bd23c4477af6a844e386e16a43a5e2ab6053e30"
Nov 01 01:20:44.988322 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:20:44.988281    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:20:54.305260 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.305224    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:20:54.386807 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.386775    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"]
Nov 01 01:20:54.387144 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.387119    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" containerName="revoke-keys"
Nov 01 01:20:54.387197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.387149    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" containerName="revoke-keys"
Nov 01 01:20:54.387243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.387218    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" containerName="revoke-keys"
Nov 01 01:20:54.390201 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.390171    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463467    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463668 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463507    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463695    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463736    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463816 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463766    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3811c0a7-1656-40ee-b264-2bb5a065ed92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463902    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.463958 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463933    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mln\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-kube-api-access-b4mln\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.464012 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.463963    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.464051 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.464042    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.502966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.502934    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"]
Nov 01 01:20:54.565736 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.565736 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565688    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mln\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-kube-api-access-b4mln\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.565736 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565728    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565768    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565824    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565859    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565900    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565927    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.565949    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3811c0a7-1656-40ee-b264-2bb5a065ed92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.566124    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.566267    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.566382    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566451 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.566443    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.566868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.566844    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3811c0a7-1656-40ee-b264-2bb5a065ed92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.568264 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.568243    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.568490 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.568473    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.627269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.627235    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.627431 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.627294    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mln\" (UniqueName: \"kubernetes.io/projected/3811c0a7-1656-40ee-b264-2bb5a065ed92-kube-api-access-b4mln\") pod \"maas-default-gateway-openshift-default-67bf986f99-6vs77\" (UID: \"3811c0a7-1656-40ee-b264-2bb5a065ed92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:54.703217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:54.703186    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:55.131851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.131817    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"]
Nov 01 01:20:55.135190 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:20:55.135163    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3811c0a7_1656_40ee_b264_2bb5a065ed92.slice/crio-cbb9a58474dfc78f7834a83f85bccc873a463af2edefe8af8f655a88dfa1014e WatchSource:0}: Error finding container cbb9a58474dfc78f7834a83f85bccc873a463af2edefe8af8f655a88dfa1014e: Status 404 returned error can't find the container with id cbb9a58474dfc78f7834a83f85bccc873a463af2edefe8af8f655a88dfa1014e
Nov 01 01:20:55.138022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.137982    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:20:55.138112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.138076    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:20:55.138200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.138121    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:20:55.905726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.905695    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" event={"ID":"3811c0a7-1656-40ee-b264-2bb5a065ed92","Type":"ContainerStarted","Data":"047bf5bfa89b64823803e774d63faa57d305e7a91505c4afd2c7ac3fa4986122"}
Nov 01 01:20:55.906173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:55.905745    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" event={"ID":"3811c0a7-1656-40ee-b264-2bb5a065ed92","Type":"ContainerStarted","Data":"cbb9a58474dfc78f7834a83f85bccc873a463af2edefe8af8f655a88dfa1014e"}
Nov 01 01:20:56.076387 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:56.076319    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podStartSLOduration=2.076297851 podStartE2EDuration="2.076297851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:20:56.075717042 +0000 UTC m=+3445.753303987" watchObservedRunningTime="2026-11-01 01:20:56.076297851 +0000 UTC m=+3445.753884796"
Nov 01 01:20:56.703735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:56.703701    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:20:56.705051 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:56.705022    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:20:56.705195 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:56.705080    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:20:57.704438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:57.704401    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:20:57.704848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:57.704461    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:20:58.704213 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:58.704178    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:20:58.704370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:58.704229    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:20:59.704352 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:59.704318    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:20:59.704843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:20:59.704390    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:21:00.703721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:00.703687    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:21:00.703904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:00.703738    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:21:01.704290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:01.704247    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 01:21:01.704704 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:01.704320    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 01:21:03.704844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:03.704796    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:21:03.705308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:03.704870    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:21:04.170975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:04.170941    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:21:04.745348 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:04.745303    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:21:04.745831 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:04.745372    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:21:05.341446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:05.341416    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:21:05.341717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:05.341693    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:21:05.342338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:05.342321    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77"
Nov 01 01:21:09.506211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.506174    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"]
Nov 01 01:21:09.509574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.509557    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.525846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.525809    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.525997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.525866    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.525997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.525978    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526020    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526048    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpcw\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-kube-api-access-5zpcw\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526171    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526214 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526202    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526284 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526274    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db8044-5359-45ce-8f45-6fc263a69edb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.526380 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.526367    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.556443 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.556408    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"]
Nov 01 01:21:09.559746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.559721    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.622649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.622610    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"]
Nov 01 01:21:09.630800 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.630766    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.631032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.630831    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.631032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.630861    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.631362 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631333    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.631726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631683    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpcw\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-kube-api-access-5zpcw\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.631841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631809    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631855    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631900    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631925    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.631975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632015    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db8044-5359-45ce-8f45-6fc263a69edb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632090 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632058    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddsv\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-kube-api-access-zddsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632091    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632152    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632181    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632222    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632264    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.632734 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632660    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632917 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632896    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.632992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632943    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db8044-5359-45ce-8f45-6fc263a69edb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.633062 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.632993    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.633062 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.633024    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.633408 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.633387    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.634584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.634559    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.636014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.635995    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.672808 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.672775    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"]
Nov 01 01:21:09.693649 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.693614    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.693797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.693712    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpcw\" (UniqueName: \"kubernetes.io/projected/d0db8044-5359-45ce-8f45-6fc263a69edb-kube-api-access-5zpcw\") pod \"maas-default-gateway-openshift-default-67bf986f99-qt8tv\" (UID: \"d0db8044-5359-45ce-8f45-6fc263a69edb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.734480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734450    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734509    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734535    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734575    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734618    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddsv\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-kube-api-access-zddsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734650    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734683    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734707    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.734908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.734751    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.735061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.735012    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.735061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.735027    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.735216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.735089    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.735288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.735263    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.735626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.735601    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.737245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.737222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.737334 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.737252    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.796082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.796007    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.796082 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.796049    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddsv\" (UniqueName: \"kubernetes.io/projected/22faf631-2efa-45a5-b54b-db37f6ef4c3a-kube-api-access-zddsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-6phck\" (UID: \"22faf631-2efa-45a5-b54b-db37f6ef4c3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:09.822047 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.822019    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:09.872208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:09.871958    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:10.247350 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.247321    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"]
Nov 01 01:21:10.249186 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:21:10.249161    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0db8044_5359_45ce_8f45_6fc263a69edb.slice/crio-52494fb4216c45c1465019d4518f63c4ca641adf8c786a1f0f3930fa99a07982 WatchSource:0}: Error finding container 52494fb4216c45c1465019d4518f63c4ca641adf8c786a1f0f3930fa99a07982: Status 404 returned error can't find the container with id 52494fb4216c45c1465019d4518f63c4ca641adf8c786a1f0f3930fa99a07982
Nov 01 01:21:10.251381 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.251347    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:10.251448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.251416    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:10.251482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.251447    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:10.301458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.301435    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"]
Nov 01 01:21:10.302589 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:21:10.302566    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22faf631_2efa_45a5_b54b_db37f6ef4c3a.slice/crio-e1d60e3393b5c591fe7537ed7122140e87ec6e2587594d3ea6f5bbce24178f83 WatchSource:0}: Error finding container e1d60e3393b5c591fe7537ed7122140e87ec6e2587594d3ea6f5bbce24178f83: Status 404 returned error can't find the container with id e1d60e3393b5c591fe7537ed7122140e87ec6e2587594d3ea6f5bbce24178f83
Nov 01 01:21:10.304876 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.304841    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:10.304988 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.304908    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:10.304988 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:10.304940    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:21:11.012613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.012565    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" event={"ID":"22faf631-2efa-45a5-b54b-db37f6ef4c3a","Type":"ContainerStarted","Data":"75ad8cfea0495f9a45c80b6d78eacd9048a439e3dc352e3cd58989142c444030"}
Nov 01 01:21:11.013064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.012635    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" event={"ID":"22faf631-2efa-45a5-b54b-db37f6ef4c3a","Type":"ContainerStarted","Data":"e1d60e3393b5c591fe7537ed7122140e87ec6e2587594d3ea6f5bbce24178f83"}
Nov 01 01:21:11.014110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.014084    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" event={"ID":"d0db8044-5359-45ce-8f45-6fc263a69edb","Type":"ContainerStarted","Data":"e8f6deb808cbc3df2d6f2afa9fa6f8641562584a4a23d123f3b67cd1715a2589"}
Nov 01 01:21:11.014263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.014123    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" event={"ID":"d0db8044-5359-45ce-8f45-6fc263a69edb","Type":"ContainerStarted","Data":"52494fb4216c45c1465019d4518f63c4ca641adf8c786a1f0f3930fa99a07982"}
Nov 01 01:21:11.143159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.136575    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podStartSLOduration=2.136555666 podStartE2EDuration="2.136555666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:21:11.133731337 +0000 UTC m=+3460.811318307" watchObservedRunningTime="2026-11-01 01:21:11.136555666 +0000 UTC m=+3460.814142612"
Nov 01 01:21:11.828219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.827204    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:11.828219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.827961    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:11.828219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.828014    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:11.872660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.872624    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:11.873939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.873921    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:11.874001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:11.873959    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:12.822591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:12.822554    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:12.823228 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:12.823191    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:12.873373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:12.873337    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:12.873603 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:12.873583    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:13.505084 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.505029    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podStartSLOduration=4.505015397 podStartE2EDuration="4.505015397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:21:11.261711227 +0000 UTC m=+3460.939298173" watchObservedRunningTime="2026-11-01 01:21:13.505015397 +0000 UTC m=+3463.182602342"
Nov 01 01:21:13.506523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.506499    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"]
Nov 01 01:21:13.510073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.510056    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.563588 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.563555    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:21:13.563874 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.563857    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:21:13.564035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.564022    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:21:13.587896 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.587863    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcsk\" (UniqueName: \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.588102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.588089    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.588539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.588503    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.665158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.647410    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:21:13.665158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.651079    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"]
Nov 01 01:21:13.689040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.689002    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.689253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.689052    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcsk\" (UniqueName: \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.689253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.689171    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.689751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.689728    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.694735 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.694695    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.771200 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.771100    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcsk\" (UniqueName: \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\") pod \"maas-api-revoke-sub-2e1f5b5f-zjqpk\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.839162 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.835636    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:13.839162 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.835701    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:13.839162 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.837221    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:13.873060 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.873030    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:13.873194 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:13.873086    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:14.252210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:14.248603    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"]
Nov 01 01:21:14.824116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:14.823647    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:14.824116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:14.823730    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:14.872455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:14.872350    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:14.872455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:14.872409    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:14.991656 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:21:14.991614    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:21:15.082040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.082002    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk" event={"ID":"e019e836-9527-4a8e-8019-5ff75f6af161","Type":"ContainerStarted","Data":"566dc9efbdc63a03533a4aca88fb2fdc021748c841544d643e3150dc0dde063e"}
Nov 01 01:21:15.082273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.082056    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk" event={"ID":"e019e836-9527-4a8e-8019-5ff75f6af161","Type":"ContainerStarted","Data":"e0ee8aadf0c4408c75a945e66706d511633b97ed74c86dd3f60853697d64c813"}
Nov 01 01:21:15.201812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.201758    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk" podStartSLOduration=2.2017414459999998 podStartE2EDuration="2.201741446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:21:15.200497721 +0000 UTC m=+3464.878084681" watchObservedRunningTime="2026-11-01 01:21:15.201741446 +0000 UTC m=+3464.879328432"
Nov 01 01:21:15.827158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.824615    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:15.827158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.824694    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:15.872655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.872620    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:15.873055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:15.872674    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:16.088714 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.088631    4493 generic.go:356] "Generic (PLEG): container finished" podID="e019e836-9527-4a8e-8019-5ff75f6af161" containerID="566dc9efbdc63a03533a4aca88fb2fdc021748c841544d643e3150dc0dde063e" exitCode=0
Nov 01 01:21:16.088853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.088750    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk" event={"ID":"e019e836-9527-4a8e-8019-5ff75f6af161","Type":"ContainerDied","Data":"566dc9efbdc63a03533a4aca88fb2fdc021748c841544d643e3150dc0dde063e"}
Nov 01 01:21:16.823608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.823568    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:16.823809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.823637    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:16.873269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.873230    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:16.873661 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:16.873298    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:17.228442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.228401    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:17.339562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.339509    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\" (UniqueName: \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\") pod \"e019e836-9527-4a8e-8019-5ff75f6af161\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") "
Nov 01 01:21:17.339753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.339616    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\") pod \"e019e836-9527-4a8e-8019-5ff75f6af161\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") "
Nov 01 01:21:17.339753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.339669    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\") pod \"e019e836-9527-4a8e-8019-5ff75f6af161\" (UID: \"e019e836-9527-4a8e-8019-5ff75f6af161\") "
Nov 01 01:21:17.340080 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.340053    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca" pod "e019e836-9527-4a8e-8019-5ff75f6af161" (UID: "e019e836-9527-4a8e-8019-5ff75f6af161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:21:17.341891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.341860    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk" pod "e019e836-9527-4a8e-8019-5ff75f6af161" (UID: "e019e836-9527-4a8e-8019-5ff75f6af161"). InnerVolumeSpecName "kube-api-access-bhcsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:21:17.341992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.341960    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca" pod "e019e836-9527-4a8e-8019-5ff75f6af161" (UID: "e019e836-9527-4a8e-8019-5ff75f6af161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:21:17.440657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.440618    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcsk\" (UniqueName: \"kubernetes.io/projected/e019e836-9527-4a8e-8019-5ff75f6af161-kube-api-access-bhcsk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:17.440657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.440651    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:17.440657 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.440661    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e019e836-9527-4a8e-8019-5ff75f6af161-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:17.823105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.823063    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qt8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 01:21:17.823323 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.823149    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv" podUID="d0db8044-5359-45ce-8f45-6fc263a69edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 01:21:17.872756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.872722    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6phck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 01:21:17.872910 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:17.872775    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck" podUID="22faf631-2efa-45a5-b54b-db37f6ef4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 01:21:18.103479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.103385    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"
Nov 01 01:21:18.103979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.103388    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk" event={"ID":"e019e836-9527-4a8e-8019-5ff75f6af161","Type":"ContainerDied","Data":"e0ee8aadf0c4408c75a945e66706d511633b97ed74c86dd3f60853697d64c813"}
Nov 01 01:21:18.103979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.103571    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ee8aadf0c4408c75a945e66706d511633b97ed74c86dd3f60853697d64c813"
Nov 01 01:21:18.826573 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.826537    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:18.826969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.826948    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:18.827717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.827699    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qt8tv"
Nov 01 01:21:18.876269 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.876238    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:18.877052 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.876990    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:18.877879 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:18.877865    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6phck"
Nov 01 01:21:26.022315 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.022285    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"]
Nov 01 01:21:26.022853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.022635    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e019e836-9527-4a8e-8019-5ff75f6af161" containerName="revoke-keys"
Nov 01 01:21:26.022853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.022646    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e019e836-9527-4a8e-8019-5ff75f6af161" containerName="revoke-keys"
Nov 01 01:21:26.022853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.022725    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e019e836-9527-4a8e-8019-5ff75f6af161" containerName="revoke-keys"
Nov 01 01:21:26.025720 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.025705    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.078438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.078407    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:21:26.078438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.078402    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:21:26.122892 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.122857    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrnh\" (UniqueName: \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.123088 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.122990    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.123088 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.123069    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.128437 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.128420    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:21:26.128529 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.128446    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:21:26.130285 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.130267    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"]
Nov 01 01:21:26.224475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.224441    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.224653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.224501    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.224653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.224559    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrnh\" (UniqueName: \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.225308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.225284    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.226966 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.226948    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.286304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.286239    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrnh\" (UniqueName: \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\") pod \"maas-api-revoke-sub-ec843018-9clp2\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.339212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.339181    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:26.401238 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.401205    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:21:26.712343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:26.712317    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"]
Nov 01 01:21:26.715193 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:21:26.715163    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16777a7b_79b8_4c26_9384_bcdc5ae0c0e3.slice/crio-cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f WatchSource:0}: Error finding container cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f: Status 404 returned error can't find the container with id cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f
Nov 01 01:21:27.151820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:27.151783    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2" event={"ID":"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3","Type":"ContainerStarted","Data":"cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f"}
Nov 01 01:21:28.184914 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:28.184417    4493 generic.go:356] "Generic (PLEG): container finished" podID="16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" containerID="4b57c1610d0bbed8a765e77a692c6b283f1b65cb84293b037069a0126b9cca5c" exitCode=0
Nov 01 01:21:28.184914 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:28.184488    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2" event={"ID":"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3","Type":"ContainerDied","Data":"4b57c1610d0bbed8a765e77a692c6b283f1b65cb84293b037069a0126b9cca5c"}
Nov 01 01:21:29.376288 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.376259    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:29.495184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.495125    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\") pod \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") "
Nov 01 01:21:29.495403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.495203    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\") pod \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") "
Nov 01 01:21:29.495403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.495245    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\" (UniqueName: \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\") pod \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\" (UID: \"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3\") "
Nov 01 01:21:29.496086 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.496060    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca" pod "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" (UID: "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:21:29.497862 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.497826    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca" pod "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" (UID: "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:21:29.498066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.498002    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh" pod "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" (UID: "16777a7b-79b8-4c26-9384-bcdc5ae0c0e3"). InnerVolumeSpecName "kube-api-access-xtrnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:21:29.596944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.596906    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:29.596944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.596937    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:29.596944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:29.596949    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrnh\" (UniqueName: \"kubernetes.io/projected/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3-kube-api-access-xtrnh\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:21:30.195692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:30.195658    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2" event={"ID":"16777a7b-79b8-4c26-9384-bcdc5ae0c0e3","Type":"ContainerDied","Data":"cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f"}
Nov 01 01:21:30.195692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:30.195677    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"
Nov 01 01:21:30.195906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:30.195709    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa35ee2e4cfbef12b43388b6b7c2d539810562dba6656fd3954d65f2178f42f"
Nov 01 01:21:44.989779 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:21:44.989738    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:21:45.303944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:21:45.303865    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:22:05.665238 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.665197    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"]
Nov 01 01:22:05.665769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.665739    4493 cpu_manager.go:414] "containerMap: removing container" podUID="16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" containerName="revoke-keys"
Nov 01 01:22:05.665769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.665758    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" containerName="revoke-keys"
Nov 01 01:22:05.665891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.665878    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" containerName="revoke-keys"
Nov 01 01:22:05.668738 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.668718    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.673032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.672995    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.673158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.673104    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.673158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.673125    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtc8s\" (UniqueName: \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.722088 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.722062    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:22:05.772049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.772013    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:22:05.772049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.772037    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:22:05.772049 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.772042    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:22:05.773589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.773569    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.773645 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.773599    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtc8s\" (UniqueName: \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.773690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.773649    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.774264 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.774248    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"]
Nov 01 01:22:05.774370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.774354    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.776358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.776340    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.835083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.835044    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtc8s\" (UniqueName: \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\") pod \"maas-api-revoke-sub-046e1343-mlcn5\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:05.982206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:05.982177    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:06.358951 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:06.358868    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"]
Nov 01 01:22:06.364065 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:22:06.364039    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294f5e67_53c2_4f51_b519_4fdcecb5e196.slice/crio-20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1 WatchSource:0}: Error finding container 20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1: Status 404 returned error can't find the container with id 20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1
Nov 01 01:22:06.377503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:06.377470    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5" event={"ID":"294f5e67-53c2-4f51-b519-4fdcecb5e196","Type":"ContainerStarted","Data":"20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1"}
Nov 01 01:22:07.383235 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:07.383198    4493 generic.go:356] "Generic (PLEG): container finished" podID="294f5e67-53c2-4f51-b519-4fdcecb5e196" containerID="98aac924359e089fdcead6378da47970c487abd2861e7c87ba3a262358ff051a" exitCode=0
Nov 01 01:22:07.383699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:07.383282    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5" event={"ID":"294f5e67-53c2-4f51-b519-4fdcecb5e196","Type":"ContainerDied","Data":"98aac924359e089fdcead6378da47970c487abd2861e7c87ba3a262358ff051a"}
Nov 01 01:22:08.528613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.528584    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:08.605434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.605401    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\" (UniqueName: \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\") pod \"294f5e67-53c2-4f51-b519-4fdcecb5e196\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") "
Nov 01 01:22:08.605608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.605470    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\") pod \"294f5e67-53c2-4f51-b519-4fdcecb5e196\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") "
Nov 01 01:22:08.605608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.605558    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\") pod \"294f5e67-53c2-4f51-b519-4fdcecb5e196\" (UID: \"294f5e67-53c2-4f51-b519-4fdcecb5e196\") "
Nov 01 01:22:08.605888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.605858    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca" pod "294f5e67-53c2-4f51-b519-4fdcecb5e196" (UID: "294f5e67-53c2-4f51-b519-4fdcecb5e196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:22:08.606001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.605978    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:08.607772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.607742    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca" pod "294f5e67-53c2-4f51-b519-4fdcecb5e196" (UID: "294f5e67-53c2-4f51-b519-4fdcecb5e196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:22:08.607912 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.607891    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s" pod "294f5e67-53c2-4f51-b519-4fdcecb5e196" (UID: "294f5e67-53c2-4f51-b519-4fdcecb5e196"). InnerVolumeSpecName "kube-api-access-rtc8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:22:08.707004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.706908    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtc8s\" (UniqueName: \"kubernetes.io/projected/294f5e67-53c2-4f51-b519-4fdcecb5e196-kube-api-access-rtc8s\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:08.707004 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:08.706944    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294f5e67-53c2-4f51-b519-4fdcecb5e196-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:09.395239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:09.395200    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5" event={"ID":"294f5e67-53c2-4f51-b519-4fdcecb5e196","Type":"ContainerDied","Data":"20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1"}
Nov 01 01:22:09.395239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:09.395228    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"
Nov 01 01:22:09.395479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:09.395261    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a49292ffeb55be5aae60738e83dc78991edbda8b7260b83f3d1291cad174a1"
Nov 01 01:22:14.988578 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:22:14.988543    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:22:16.430345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.430311    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"]
Nov 01 01:22:16.430819 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.430702    4493 cpu_manager.go:414] "containerMap: removing container" podUID="294f5e67-53c2-4f51-b519-4fdcecb5e196" containerName="revoke-keys"
Nov 01 01:22:16.430819 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.430713    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f5e67-53c2-4f51-b519-4fdcecb5e196" containerName="revoke-keys"
Nov 01 01:22:16.430819 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.430785    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f5e67-53c2-4f51-b519-4fdcecb5e196" containerName="revoke-keys"
Nov 01 01:22:16.433889 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.433873    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.476860 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.476828    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztvn\" (UniqueName: \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.477040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.476969    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.477099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.477036    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.536522 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.536488    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:22:16.536694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.536549    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:22:16.536694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.536552    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:22:16.536694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.536602    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:22:16.538856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.538838    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"]
Nov 01 01:22:16.577759 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.577726    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.577924 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.577781    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.577924 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.577878    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztvn\" (UniqueName: \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.578539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.578515    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.580322 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.580304    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.639845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.639815    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztvn\" (UniqueName: \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\") pod \"maas-api-revoke-sub-d3fba056-xfrb7\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:16.746826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:16.746793    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:17.120345 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:17.120318    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"]
Nov 01 01:22:17.122424 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:22:17.122396    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c41a569_ca9c_489c_8652_5a665589f13b.slice/crio-97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf WatchSource:0}: Error finding container 97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf: Status 404 returned error can't find the container with id 97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf
Nov 01 01:22:17.433048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:17.432958    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7" event={"ID":"6c41a569-ca9c-489c-8652-5a665589f13b","Type":"ContainerStarted","Data":"0c41a514eaf2675eb30e6cb7d2e2d4c0b16867e8ae3e00a4a8c2e1d4e0d53c69"}
Nov 01 01:22:17.433048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:17.433015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7" event={"ID":"6c41a569-ca9c-489c-8652-5a665589f13b","Type":"ContainerStarted","Data":"97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf"}
Nov 01 01:22:17.550647 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:17.550593    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7" podStartSLOduration=1.550578344 podStartE2EDuration="1.550578344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:22:17.548838803 +0000 UTC m=+3527.226425743" watchObservedRunningTime="2026-11-01 01:22:17.550578344 +0000 UTC m=+3527.228165289"
Nov 01 01:22:18.439100 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:18.439069    4493 generic.go:356] "Generic (PLEG): container finished" podID="6c41a569-ca9c-489c-8652-5a665589f13b" containerID="0c41a514eaf2675eb30e6cb7d2e2d4c0b16867e8ae3e00a4a8c2e1d4e0d53c69" exitCode=0
Nov 01 01:22:18.439614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:18.439170    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7" event={"ID":"6c41a569-ca9c-489c-8652-5a665589f13b","Type":"ContainerDied","Data":"0c41a514eaf2675eb30e6cb7d2e2d4c0b16867e8ae3e00a4a8c2e1d4e0d53c69"}
Nov 01 01:22:18.611840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:18.611807    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:22:19.644369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.644339    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:19.707528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.707437    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\" (UniqueName: \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\") pod \"6c41a569-ca9c-489c-8652-5a665589f13b\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") "
Nov 01 01:22:19.707781 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.707753    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\") pod \"6c41a569-ca9c-489c-8652-5a665589f13b\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") "
Nov 01 01:22:19.707932 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.707819    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\") pod \"6c41a569-ca9c-489c-8652-5a665589f13b\" (UID: \"6c41a569-ca9c-489c-8652-5a665589f13b\") "
Nov 01 01:22:19.708340 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.708297    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca" pod "6c41a569-ca9c-489c-8652-5a665589f13b" (UID: "6c41a569-ca9c-489c-8652-5a665589f13b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:22:19.710906 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.710868    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn" pod "6c41a569-ca9c-489c-8652-5a665589f13b" (UID: "6c41a569-ca9c-489c-8652-5a665589f13b"). InnerVolumeSpecName "kube-api-access-gztvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:22:19.711112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.711087    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca" pod "6c41a569-ca9c-489c-8652-5a665589f13b" (UID: "6c41a569-ca9c-489c-8652-5a665589f13b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:22:19.809299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.809213    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztvn\" (UniqueName: \"kubernetes.io/projected/6c41a569-ca9c-489c-8652-5a665589f13b-kube-api-access-gztvn\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:19.809299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.809256    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:19.809299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:19.809272    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41a569-ca9c-489c-8652-5a665589f13b-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:20.469158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:20.468904    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7" event={"ID":"6c41a569-ca9c-489c-8652-5a665589f13b","Type":"ContainerDied","Data":"97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf"}
Nov 01 01:22:20.469158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:20.468971    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97cddedbfebf4dc0a4e94347f15862f0b906e57ed7bb3df8764e31dab76a0dcf"
Nov 01 01:22:20.469158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:20.469043    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"
Nov 01 01:22:21.393158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:21.392309    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:22:21.393158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:21.392394    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:22:29.276979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:29.276944    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:22:41.981406 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:41.981321    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:22:44.989108 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:22:44.989073    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:22:54.522527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.522490    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"]
Nov 01 01:22:54.522995 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.522873    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6c41a569-ca9c-489c-8652-5a665589f13b" containerName="revoke-keys"
Nov 01 01:22:54.522995 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.522887    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c41a569-ca9c-489c-8652-5a665589f13b" containerName="revoke-keys"
Nov 01 01:22:54.522995 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.522954    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c41a569-ca9c-489c-8652-5a665589f13b" containerName="revoke-keys"
Nov 01 01:22:54.526021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.525997    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.578479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.578431    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:22:54.578479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.578476    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:22:54.604910 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.604877    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.605103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.605077    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krtp\" (UniqueName: \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.605193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.605118    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.628479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.628450    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:22:54.628479 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.628457    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:22:54.630573 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.630553    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"]
Nov 01 01:22:54.705936 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.705908    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krtp\" (UniqueName: \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.706122 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.705943    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.706246 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.706156    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.706549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.706527    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.708752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.708733    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.767074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.767042    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krtp\" (UniqueName: \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\") pod \"maas-api-revoke-sub-ae594979-xs4ph\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:54.839741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:54.839654    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:55.212234 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:55.212211    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"]
Nov 01 01:22:55.214438 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:22:55.214405    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d9abba_db0c_471a_b69f_e6d6c4d2bfdf.slice/crio-9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1 WatchSource:0}: Error finding container 9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1: Status 404 returned error can't find the container with id 9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1
Nov 01 01:22:55.671317 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:55.671278    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph" event={"ID":"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf","Type":"ContainerStarted","Data":"eeace95fe80c463039aa5e3dc7dd979e018cc891a8167c1393145c4803e539a5"}
Nov 01 01:22:55.671809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:55.671355    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph" event={"ID":"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf","Type":"ContainerStarted","Data":"9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1"}
Nov 01 01:22:55.788414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:55.788303    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph" podStartSLOduration=1.788289449 podStartE2EDuration="1.788289449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:22:55.786926699 +0000 UTC m=+3565.464513636" watchObservedRunningTime="2026-11-01 01:22:55.788289449 +0000 UTC m=+3565.465876452"
Nov 01 01:22:56.677578 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:56.677540    4493 generic.go:356] "Generic (PLEG): container finished" podID="77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" containerID="eeace95fe80c463039aa5e3dc7dd979e018cc891a8167c1393145c4803e539a5" exitCode=0
Nov 01 01:22:56.678022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:56.677601    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph" event={"ID":"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf","Type":"ContainerDied","Data":"eeace95fe80c463039aa5e3dc7dd979e018cc891a8167c1393145c4803e539a5"}
Nov 01 01:22:57.813926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.813901    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:57.936123 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.936047    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\") pod \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") "
Nov 01 01:22:57.936302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.936199    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\" (UniqueName: \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\") pod \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") "
Nov 01 01:22:57.936302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.936230    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\") pod \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\" (UID: \"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf\") "
Nov 01 01:22:57.936775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.936741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca" pod "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" (UID: "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:22:57.938550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.938520    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp" pod "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" (UID: "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf"). InnerVolumeSpecName "kube-api-access-2krtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:22:57.938878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:57.938857    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca" pod "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" (UID: "77d9abba-db0c-471a-b69f-e6d6c4d2bfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:22:58.037570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.037527    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2krtp\" (UniqueName: \"kubernetes.io/projected/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-kube-api-access-2krtp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:58.037570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.037561    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:58.037570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.037571    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:22:58.689161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.689118    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"
Nov 01 01:22:58.689370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.689162    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph" event={"ID":"77d9abba-db0c-471a-b69f-e6d6c4d2bfdf","Type":"ContainerDied","Data":"9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1"}
Nov 01 01:22:58.689370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:22:58.689208    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a16990495434501e06d8a95190e8bad91f5edb0ff3684490c4e00e9994389e1"
Nov 01 01:23:07.491594 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:07.491558    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:23:15.103366 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:23:15.103193    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:23:17.086481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:17.086439    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:23:22.700732 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.700691    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"]
Nov 01 01:23:22.702190 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.702162    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" containerName="revoke-keys"
Nov 01 01:23:22.702357 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.702344    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" containerName="revoke-keys"
Nov 01 01:23:22.702697 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.702682    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" containerName="revoke-keys"
Nov 01 01:23:22.707254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.707227    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.759753 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.759715    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"]
Nov 01 01:23:22.807091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.807053    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:23:22.807091 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.807075    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:23:22.807324 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.807101    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:23:22.809480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.809463    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:23:22.829438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.829407    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.829597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.829454    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.829597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.829517    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tvw\" (UniqueName: \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.930180 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.930127    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.930360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.930209    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.930360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.930268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tvw\" (UniqueName: \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.930824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.930802    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.932716 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.932698    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:22.991869 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:22.991842    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tvw\" (UniqueName: \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\") pod \"maas-api-revoke-sub-a4aef7d7-dj6j5\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:23.020837 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:23.020810    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:23.394820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:23.394798    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"]
Nov 01 01:23:23.396919 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:23:23.396885    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418e7eab_301e_4bdd_8dc9_d1bc26055629.slice/crio-a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a WatchSource:0}: Error finding container a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a: Status 404 returned error can't find the container with id a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a
Nov 01 01:23:23.861739 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:23.861702    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5" event={"ID":"418e7eab-301e-4bdd-8dc9-d1bc26055629","Type":"ContainerStarted","Data":"84e2e35b1cb43f26380d9ed27ddbabe031a4af4205f7ce9d249a5603f8828694"}
Nov 01 01:23:23.862185 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:23.861756    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5" event={"ID":"418e7eab-301e-4bdd-8dc9-d1bc26055629","Type":"ContainerStarted","Data":"a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a"}
Nov 01 01:23:23.978944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:23.978843    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5" podStartSLOduration=1.978828512 podStartE2EDuration="1.978828512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:23:23.977366292 +0000 UTC m=+3593.654953250" watchObservedRunningTime="2026-11-01 01:23:23.978828512 +0000 UTC m=+3593.656415507"
Nov 01 01:23:24.868193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:24.868161    4493 generic.go:356] "Generic (PLEG): container finished" podID="418e7eab-301e-4bdd-8dc9-d1bc26055629" containerID="84e2e35b1cb43f26380d9ed27ddbabe031a4af4205f7ce9d249a5603f8828694" exitCode=0
Nov 01 01:23:24.868577 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:24.868250    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5" event={"ID":"418e7eab-301e-4bdd-8dc9-d1bc26055629","Type":"ContainerDied","Data":"84e2e35b1cb43f26380d9ed27ddbabe031a4af4205f7ce9d249a5603f8828694"}
Nov 01 01:23:26.007247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.007224    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:26.061197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.061166    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\") pod \"418e7eab-301e-4bdd-8dc9-d1bc26055629\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") "
Nov 01 01:23:26.061321 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.061289    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\") pod \"418e7eab-301e-4bdd-8dc9-d1bc26055629\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") "
Nov 01 01:23:26.061387 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.061372    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\" (UniqueName: \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\") pod \"418e7eab-301e-4bdd-8dc9-d1bc26055629\" (UID: \"418e7eab-301e-4bdd-8dc9-d1bc26055629\") "
Nov 01 01:23:26.061593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.061572    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca" pod "418e7eab-301e-4bdd-8dc9-d1bc26055629" (UID: "418e7eab-301e-4bdd-8dc9-d1bc26055629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:23:26.061645 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.061633    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:26.063505 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.063479    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca" pod "418e7eab-301e-4bdd-8dc9-d1bc26055629" (UID: "418e7eab-301e-4bdd-8dc9-d1bc26055629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:23:26.063565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.063524    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw" pod "418e7eab-301e-4bdd-8dc9-d1bc26055629" (UID: "418e7eab-301e-4bdd-8dc9-d1bc26055629"). InnerVolumeSpecName "kube-api-access-t7tvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:23:26.162349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.162264    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418e7eab-301e-4bdd-8dc9-d1bc26055629-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:26.162349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.162295    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tvw\" (UniqueName: \"kubernetes.io/projected/418e7eab-301e-4bdd-8dc9-d1bc26055629-kube-api-access-t7tvw\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:26.878513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.878429    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"
Nov 01 01:23:26.878513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.878463    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5" event={"ID":"418e7eab-301e-4bdd-8dc9-d1bc26055629","Type":"ContainerDied","Data":"a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a"}
Nov 01 01:23:26.878513 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:26.878511    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a155d8016a809d9594c7cc7648abefad515c689e86554b296e2d2bb8d1d2620a"
Nov 01 01:23:35.691992 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:35.691952    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:23:40.050694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:40.050656    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"]
Nov 01 01:23:40.100695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:40.100664    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c93b493-d8crr"]
Nov 01 01:23:40.502078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:40.502046    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fe6766-0d13-4a66-920f-366546492797" path="/var/lib/kubelet/pods/a5fe6766-0d13-4a66-920f-366546492797/volumes"
Nov 01 01:23:45.104896 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:23:45.104749    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:23:48.260398 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:48.260350    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:23:54.060699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.060668    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"]
Nov 01 01:23:54.061184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.061057    4493 cpu_manager.go:414] "containerMap: removing container" podUID="418e7eab-301e-4bdd-8dc9-d1bc26055629" containerName="revoke-keys"
Nov 01 01:23:54.061184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.061070    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418e7eab-301e-4bdd-8dc9-d1bc26055629" containerName="revoke-keys"
Nov 01 01:23:54.061184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.061150    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="418e7eab-301e-4bdd-8dc9-d1bc26055629" containerName="revoke-keys"
Nov 01 01:23:54.064514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.064491    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.117065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.117038    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:23:54.117251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.117080    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:23:54.157699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.157668    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtsbh\" (UniqueName: \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.157891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.157712    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.157891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.157868    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.166973 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.166945    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:23:54.167101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.166980    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:23:54.169169 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.169151    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"]
Nov 01 01:23:54.258971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.258932    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtsbh\" (UniqueName: \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.259178 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.258996    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.259178 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.259114    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.259687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.259660    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.261306 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.261287    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.320299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.320231    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtsbh\" (UniqueName: \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\") pod \"maas-api-revoke-sub-79805ca9-qn8h6\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.377356 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.377321    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:54.749951 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:54.749924    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"]
Nov 01 01:23:54.752288 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:23:54.752260    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668470cd_74dd_4bcf_92f3_db9bb4754fc9.slice/crio-1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0 WatchSource:0}: Error finding container 1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0: Status 404 returned error can't find the container with id 1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0
Nov 01 01:23:55.031723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:55.031693    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6" event={"ID":"668470cd-74dd-4bcf-92f3-db9bb4754fc9","Type":"ContainerStarted","Data":"1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0"}
Nov 01 01:23:56.038250 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:56.038216    4493 generic.go:356] "Generic (PLEG): container finished" podID="668470cd-74dd-4bcf-92f3-db9bb4754fc9" containerID="44c5c31410e77703a56a563c49789d1844644e9d08c1f8ddbfa16ccc652c1272" exitCode=0
Nov 01 01:23:56.038658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:56.038309    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6" event={"ID":"668470cd-74dd-4bcf-92f3-db9bb4754fc9","Type":"ContainerDied","Data":"44c5c31410e77703a56a563c49789d1844644e9d08c1f8ddbfa16ccc652c1272"}
Nov 01 01:23:57.173467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.173445    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:57.287591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.287558    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\" (UniqueName: \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\") pod \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") "
Nov 01 01:23:57.287761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.287600    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\") pod \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") "
Nov 01 01:23:57.287761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.287646    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\") pod \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\" (UID: \"668470cd-74dd-4bcf-92f3-db9bb4754fc9\") "
Nov 01 01:23:57.288077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.288049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca" pod "668470cd-74dd-4bcf-92f3-db9bb4754fc9" (UID: "668470cd-74dd-4bcf-92f3-db9bb4754fc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:23:57.290182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.290109    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca" pod "668470cd-74dd-4bcf-92f3-db9bb4754fc9" (UID: "668470cd-74dd-4bcf-92f3-db9bb4754fc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:23:57.290364 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.290344    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh" pod "668470cd-74dd-4bcf-92f3-db9bb4754fc9" (UID: "668470cd-74dd-4bcf-92f3-db9bb4754fc9"). InnerVolumeSpecName "kube-api-access-xtsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:23:57.388977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.388941    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtsbh\" (UniqueName: \"kubernetes.io/projected/668470cd-74dd-4bcf-92f3-db9bb4754fc9-kube-api-access-xtsbh\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:57.388977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.388973    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:57.388977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:57.388985    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668470cd-74dd-4bcf-92f3-db9bb4754fc9-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:23:58.049783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:58.049752    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6" event={"ID":"668470cd-74dd-4bcf-92f3-db9bb4754fc9","Type":"ContainerDied","Data":"1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0"}
Nov 01 01:23:58.049783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:58.049773    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"
Nov 01 01:23:58.049997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:23:58.049800    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed13f9e91e483d3a32e1636e938ca64bb911bc013e8b20b6772922b30ba24d0"
Nov 01 01:24:08.774029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:08.773981    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:24:14.801357 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.801271    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"]
Nov 01 01:24:14.801835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.801813    4493 cpu_manager.go:414] "containerMap: removing container" podUID="668470cd-74dd-4bcf-92f3-db9bb4754fc9" containerName="revoke-keys"
Nov 01 01:24:14.801918 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.801839    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668470cd-74dd-4bcf-92f3-db9bb4754fc9" containerName="revoke-keys"
Nov 01 01:24:14.801976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.801958    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="668470cd-74dd-4bcf-92f3-db9bb4754fc9" containerName="revoke-keys"
Nov 01 01:24:14.805011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.804990    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:14.857573 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.857533    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:24:14.857751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.857607    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:24:14.857751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.857696    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:24:14.903494 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.903464    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjvr\" (UniqueName: \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:14.903673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.903600    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:14.903673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.903638    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:14.909880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.909857    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"]
Nov 01 01:24:14.957793 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:14.957765    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:24:14.988669 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:24:14.988639    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:24:15.005238 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.005124    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjvr\" (UniqueName: \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.005415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.005312    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.005415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.005347    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.006011 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.005992    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.007818 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.007801    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.067186 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.067103    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjvr\" (UniqueName: \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\") pod \"maas-api-revoke-sub-0b7e8edb-5fp49\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.118193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.118158    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:15.493563 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:15.493534    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"]
Nov 01 01:24:15.495649 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:24:15.495604    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfed82e_a73f_40dc_bec4_c5124aad95f7.slice/crio-eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6 WatchSource:0}: Error finding container eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6: Status 404 returned error can't find the container with id eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6
Nov 01 01:24:16.159035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:16.158924    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49" event={"ID":"7dfed82e-a73f-40dc-bec4-c5124aad95f7","Type":"ContainerStarted","Data":"b78ad2d07d3339686e0c2df986de6a6ffab935993c20c12e765d177b88bccd12"}
Nov 01 01:24:16.159035 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:16.158988    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49" event={"ID":"7dfed82e-a73f-40dc-bec4-c5124aad95f7","Type":"ContainerStarted","Data":"eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6"}
Nov 01 01:24:16.276409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:16.276356    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49" podStartSLOduration=2.276341463 podStartE2EDuration="2.276341463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:24:16.27465695 +0000 UTC m=+3645.952243896" watchObservedRunningTime="2026-11-01 01:24:16.276341463 +0000 UTC m=+3645.953928408"
Nov 01 01:24:17.165429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:17.165394    4493 generic.go:356] "Generic (PLEG): container finished" podID="7dfed82e-a73f-40dc-bec4-c5124aad95f7" containerID="b78ad2d07d3339686e0c2df986de6a6ffab935993c20c12e765d177b88bccd12" exitCode=0
Nov 01 01:24:17.165886 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:17.165481    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49" event={"ID":"7dfed82e-a73f-40dc-bec4-c5124aad95f7","Type":"ContainerDied","Data":"b78ad2d07d3339686e0c2df986de6a6ffab935993c20c12e765d177b88bccd12"}
Nov 01 01:24:18.303147 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.303108    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:18.435794 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.435717    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\") pod \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") "
Nov 01 01:24:18.435948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.435843    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\") pod \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") "
Nov 01 01:24:18.435948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.435909    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\" (UniqueName: \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\") pod \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\" (UID: \"7dfed82e-a73f-40dc-bec4-c5124aad95f7\") "
Nov 01 01:24:18.436318 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.436282    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca" pod "7dfed82e-a73f-40dc-bec4-c5124aad95f7" (UID: "7dfed82e-a73f-40dc-bec4-c5124aad95f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:24:18.438097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.438071    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca" pod "7dfed82e-a73f-40dc-bec4-c5124aad95f7" (UID: "7dfed82e-a73f-40dc-bec4-c5124aad95f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:24:18.438231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.438183    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr" pod "7dfed82e-a73f-40dc-bec4-c5124aad95f7" (UID: "7dfed82e-a73f-40dc-bec4-c5124aad95f7"). InnerVolumeSpecName "kube-api-access-lkjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:24:18.537466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.537424    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:18.537466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.537459    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed82e-a73f-40dc-bec4-c5124aad95f7-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:18.537466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:18.537470    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjvr\" (UniqueName: \"kubernetes.io/projected/7dfed82e-a73f-40dc-bec4-c5124aad95f7-kube-api-access-lkjvr\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:19.176609 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:19.176573    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"
Nov 01 01:24:19.176609 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:19.176593    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49" event={"ID":"7dfed82e-a73f-40dc-bec4-c5124aad95f7","Type":"ContainerDied","Data":"eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6"}
Nov 01 01:24:19.176827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:19.176647    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb1d06e923e8e73e455db91faaf749e6aa44732e63595caad7d447539b3240c6"
Nov 01 01:24:26.171594 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.171548    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"]
Nov 01 01:24:26.172245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.171953    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7dfed82e-a73f-40dc-bec4-c5124aad95f7" containerName="revoke-keys"
Nov 01 01:24:26.172245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.171969    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfed82e-a73f-40dc-bec4-c5124aad95f7" containerName="revoke-keys"
Nov 01 01:24:26.172245 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.172046    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfed82e-a73f-40dc-bec4-c5124aad95f7" containerName="revoke-keys"
Nov 01 01:24:26.175474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.175434    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.228190 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.228161    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:24:26.228375 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.228206    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:24:26.278182 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.278151    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:24:26.278360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.278207    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:24:26.280440 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.280421    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"]
Nov 01 01:24:26.322844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.322804    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.323044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.322969    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95v2z\" (UniqueName: \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.323044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.323028    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.424547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.424452    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95v2z\" (UniqueName: \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.424746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.424606    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.424746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.424665    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.425299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.425272    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.427079 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.427058    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.486453 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.486415    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95v2z\" (UniqueName: \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\") pod \"maas-api-revoke-sub-3d4156f1-dg562\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.492505 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.492481    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:26.867401 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:26.867369    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"]
Nov 01 01:24:26.870014 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:24:26.869986    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4148b1_4181_4fbd_bba6_1832b031dec8.slice/crio-f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6 WatchSource:0}: Error finding container f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6: Status 404 returned error can't find the container with id f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6
Nov 01 01:24:27.216774 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:27.216732    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562" event={"ID":"5d4148b1-4181-4fbd-bba6-1832b031dec8","Type":"ContainerStarted","Data":"0724e4f1cbfdb31e154935664b28b1a6b7dc6098b25844ae1ecfa1ae86530a95"}
Nov 01 01:24:27.217242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:27.216793    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562" event={"ID":"5d4148b1-4181-4fbd-bba6-1832b031dec8","Type":"ContainerStarted","Data":"f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6"}
Nov 01 01:24:27.334968 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:27.334915    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562" podStartSLOduration=1.3348973690000001 podStartE2EDuration="1.334897369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:24:27.333166037 +0000 UTC m=+3657.010752983" watchObservedRunningTime="2026-11-01 01:24:27.334897369 +0000 UTC m=+3657.012484315"
Nov 01 01:24:27.779561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:27.779527    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:24:28.237442 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:28.237410    4493 generic.go:356] "Generic (PLEG): container finished" podID="5d4148b1-4181-4fbd-bba6-1832b031dec8" containerID="0724e4f1cbfdb31e154935664b28b1a6b7dc6098b25844ae1ecfa1ae86530a95" exitCode=0
Nov 01 01:24:28.257273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:28.257073    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562" event={"ID":"5d4148b1-4181-4fbd-bba6-1832b031dec8","Type":"ContainerDied","Data":"0724e4f1cbfdb31e154935664b28b1a6b7dc6098b25844ae1ecfa1ae86530a95"}
Nov 01 01:24:30.034567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.034535    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:30.110374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.110332    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\" (UniqueName: \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\") pod \"5d4148b1-4181-4fbd-bba6-1832b031dec8\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") "
Nov 01 01:24:30.110374 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.110391    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\") pod \"5d4148b1-4181-4fbd-bba6-1832b031dec8\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") "
Nov 01 01:24:30.110685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.110494    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\") pod \"5d4148b1-4181-4fbd-bba6-1832b031dec8\" (UID: \"5d4148b1-4181-4fbd-bba6-1832b031dec8\") "
Nov 01 01:24:30.111099 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.111070    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca" pod "5d4148b1-4181-4fbd-bba6-1832b031dec8" (UID: "5d4148b1-4181-4fbd-bba6-1832b031dec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:24:30.112718 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.112693    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca" pod "5d4148b1-4181-4fbd-bba6-1832b031dec8" (UID: "5d4148b1-4181-4fbd-bba6-1832b031dec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:24:30.113502 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.113477    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z" pod "5d4148b1-4181-4fbd-bba6-1832b031dec8" (UID: "5d4148b1-4181-4fbd-bba6-1832b031dec8"). InnerVolumeSpecName "kube-api-access-95v2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:24:30.211543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.211502    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95v2z\" (UniqueName: \"kubernetes.io/projected/5d4148b1-4181-4fbd-bba6-1832b031dec8-kube-api-access-95v2z\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:30.211543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.211537    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:30.211543 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.211548    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4148b1-4181-4fbd-bba6-1832b031dec8-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:30.276720 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.276631    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562" event={"ID":"5d4148b1-4181-4fbd-bba6-1832b031dec8","Type":"ContainerDied","Data":"f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6"}
Nov 01 01:24:30.276720 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.276662    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"
Nov 01 01:24:30.276913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:30.276723    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5ef3408fa9dec0d6927e1dc3e0c3c4789327e975d370db29c9f9a4ef6d33da6"
Nov 01 01:24:31.793276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:31.793231    4493 scope.go:122] "RemoveContainer" containerID="354d9c9f83e4cd33f34edbfce563c17ce9596f785591d2209e3815f15b23d2c8"
Nov 01 01:24:31.801929 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:31.801908    4493 scope.go:122] "RemoveContainer" containerID="ac2e9a08cd6ce9c0854f91d8efaf979ebc18d34242a9742da3412d12c2f6da4c"
Nov 01 01:24:35.074707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:35.074668    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"]
Nov 01 01:24:35.123776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:35.123739    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tgvpv"]
Nov 01 01:24:36.500724 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:36.500679    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd0b394-0224-449f-8dfb-c8c73b31f8c7" path="/var/lib/kubelet/pods/4cd0b394-0224-449f-8dfb-c8c73b31f8c7/volumes"
Nov 01 01:24:44.217751 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.217715    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"]
Nov 01 01:24:44.267639 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.267604    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dg562"]
Nov 01 01:24:44.338385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.338347    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"]
Nov 01 01:24:44.338748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.338734    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5d4148b1-4181-4fbd-bba6-1832b031dec8" containerName="revoke-keys"
Nov 01 01:24:44.338824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.338748    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4148b1-4181-4fbd-bba6-1832b031dec8" containerName="revoke-keys"
Nov 01 01:24:44.338883 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.338833    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4148b1-4181-4fbd-bba6-1832b031dec8" containerName="revoke-keys"
Nov 01 01:24:44.341817 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.341794    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.394471 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.394437    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:24:44.394676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.394512    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:24:44.394676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.394553    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:24:44.394676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.394573    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:24:44.401206 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.401172    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"]
Nov 01 01:24:44.440301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.440265    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.440517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.440422    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.440517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.440504    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzgc\" (UniqueName: \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.504472 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.504385    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4148b1-4181-4fbd-bba6-1832b031dec8" path="/var/lib/kubelet/pods/5d4148b1-4181-4fbd-bba6-1832b031dec8/volumes"
Nov 01 01:24:44.541524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.541475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.541715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.541571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.541715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.541603    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzgc\" (UniqueName: \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.542199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.542177    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.544376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.544340    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.603414 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.603387    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzgc\" (UniqueName: \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\") pod \"maas-api-revoke-sub-3d4156f1-m7mbl\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.655956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:44.655913    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:44.988648 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:24:44.988614    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:24:45.031593 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:45.031571    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"]
Nov 01 01:24:45.033565 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:24:45.033537    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c00388_d572_48b8_b6c9_f8f439360f9c.slice/crio-5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982 WatchSource:0}: Error finding container 5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982: Status 404 returned error can't find the container with id 5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982
Nov 01 01:24:45.349757 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:45.349729    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl" event={"ID":"d8c00388-d572-48b8-b6c9-f8f439360f9c","Type":"ContainerStarted","Data":"d33627a3c220a4849ea925219cf9c3d1792ac7fa4d04b496d34b08896165ff83"}
Nov 01 01:24:45.350113 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:45.349783    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl" event={"ID":"d8c00388-d572-48b8-b6c9-f8f439360f9c","Type":"ContainerStarted","Data":"5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982"}
Nov 01 01:24:45.467282 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:45.467233    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl" podStartSLOduration=1.467218607 podStartE2EDuration="1.467218607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:24:45.46510851 +0000 UTC m=+3675.142695455" watchObservedRunningTime="2026-11-01 01:24:45.467218607 +0000 UTC m=+3675.144805552"
Nov 01 01:24:46.356211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:46.356167    4493 generic.go:356] "Generic (PLEG): container finished" podID="d8c00388-d572-48b8-b6c9-f8f439360f9c" containerID="d33627a3c220a4849ea925219cf9c3d1792ac7fa4d04b496d34b08896165ff83" exitCode=0
Nov 01 01:24:46.356211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:46.356227    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl" event={"ID":"d8c00388-d572-48b8-b6c9-f8f439360f9c","Type":"ContainerDied","Data":"d33627a3c220a4849ea925219cf9c3d1792ac7fa4d04b496d34b08896165ff83"}
Nov 01 01:24:47.496857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.496836    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:47.568325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.568295    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\" (UniqueName: \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\") pod \"d8c00388-d572-48b8-b6c9-f8f439360f9c\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") "
Nov 01 01:24:47.568465 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.568417    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\") pod \"d8c00388-d572-48b8-b6c9-f8f439360f9c\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") "
Nov 01 01:24:47.568465 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.568443    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\") pod \"d8c00388-d572-48b8-b6c9-f8f439360f9c\" (UID: \"d8c00388-d572-48b8-b6c9-f8f439360f9c\") "
Nov 01 01:24:47.568800 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.568777    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca" pod "d8c00388-d572-48b8-b6c9-f8f439360f9c" (UID: "d8c00388-d572-48b8-b6c9-f8f439360f9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:24:47.570741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.570712    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca" pod "d8c00388-d572-48b8-b6c9-f8f439360f9c" (UID: "d8c00388-d572-48b8-b6c9-f8f439360f9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:24:47.570835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.570716    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc" pod "d8c00388-d572-48b8-b6c9-f8f439360f9c" (UID: "d8c00388-d572-48b8-b6c9-f8f439360f9c"). InnerVolumeSpecName "kube-api-access-5jzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:24:47.670034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.669940    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jzgc\" (UniqueName: \"kubernetes.io/projected/d8c00388-d572-48b8-b6c9-f8f439360f9c-kube-api-access-5jzgc\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:47.670034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.669969    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:47.670034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:47.669980    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c00388-d572-48b8-b6c9-f8f439360f9c-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:24:48.366617 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:48.366579    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl" event={"ID":"d8c00388-d572-48b8-b6c9-f8f439360f9c","Type":"ContainerDied","Data":"5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982"}
Nov 01 01:24:48.366806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:48.366631    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"
Nov 01 01:24:48.366806 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:24:48.366666    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fba4b4c347a9ce57f3b1b8769fee9fbb9331b69c93d73095545033aa6b39982"
Nov 01 01:25:12.178865 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:12.178832    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:25:15.112216 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:25:15.112170    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:25:22.546076 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.546043    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"]
Nov 01 01:25:22.546626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.546602    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d8c00388-d572-48b8-b6c9-f8f439360f9c" containerName="revoke-keys"
Nov 01 01:25:22.546626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.546627    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c00388-d572-48b8-b6c9-f8f439360f9c" containerName="revoke-keys"
Nov 01 01:25:22.546805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.546782    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c00388-d572-48b8-b6c9-f8f439360f9c" containerName="revoke-keys"
Nov 01 01:25:22.550222 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.550203    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.605015 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.604984    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:25:22.605242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.605056    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:25:22.655010 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.654971    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:25:22.655209 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.655100    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:25:22.657293 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.657272    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"]
Nov 01 01:25:22.660448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.660430    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.660500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.660478    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjpk\" (UniqueName: \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.660585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.660565    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.761585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.761552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.761773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.761603    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjpk\" (UniqueName: \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.761773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.761634    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.762221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.762195    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.764097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.764079    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.823160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.823063    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjpk\" (UniqueName: \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\") pod \"maas-api-revoke-sub-e2fc9c5b-fm25d\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:22.862888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:22.862856    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:23.239184 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:23.239157    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"]
Nov 01 01:25:23.240995 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:25:23.240969    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c023af_9183_406d_a9ac_9a7e0ab2fb86.slice/crio-d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664 WatchSource:0}: Error finding container d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664: Status 404 returned error can't find the container with id d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664
Nov 01 01:25:23.556967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:23.556933    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d" event={"ID":"93c023af-9183-406d-a9ac-9a7e0ab2fb86","Type":"ContainerStarted","Data":"54bc00f95a337409f3943d6fc0cf7029d968bd3a40d3b59a9305e84957ae97c6"}
Nov 01 01:25:23.557370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:23.556991    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d" event={"ID":"93c023af-9183-406d-a9ac-9a7e0ab2fb86","Type":"ContainerStarted","Data":"d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664"}
Nov 01 01:25:23.673673 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:23.673626    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d" podStartSLOduration=1.673605345 podStartE2EDuration="1.673605345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:25:23.673238963 +0000 UTC m=+3713.350825908" watchObservedRunningTime="2026-11-01 01:25:23.673605345 +0000 UTC m=+3713.351192290"
Nov 01 01:25:24.563417 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:24.563385    4493 generic.go:356] "Generic (PLEG): container finished" podID="93c023af-9183-406d-a9ac-9a7e0ab2fb86" containerID="54bc00f95a337409f3943d6fc0cf7029d968bd3a40d3b59a9305e84957ae97c6" exitCode=0
Nov 01 01:25:24.563856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:24.563468    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d" event={"ID":"93c023af-9183-406d-a9ac-9a7e0ab2fb86","Type":"ContainerDied","Data":"54bc00f95a337409f3943d6fc0cf7029d968bd3a40d3b59a9305e84957ae97c6"}
Nov 01 01:25:25.698482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.698456    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:25.788926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.788894    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\") pod \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") "
Nov 01 01:25:25.789112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.788989    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\") pod \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") "
Nov 01 01:25:25.789112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.789033    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\" (UniqueName: \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\") pod \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\" (UID: \"93c023af-9183-406d-a9ac-9a7e0ab2fb86\") "
Nov 01 01:25:25.789403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.789373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca" pod "93c023af-9183-406d-a9ac-9a7e0ab2fb86" (UID: "93c023af-9183-406d-a9ac-9a7e0ab2fb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:25:25.791160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.791125    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca" pod "93c023af-9183-406d-a9ac-9a7e0ab2fb86" (UID: "93c023af-9183-406d-a9ac-9a7e0ab2fb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:25:25.791292 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.791273    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk" pod "93c023af-9183-406d-a9ac-9a7e0ab2fb86" (UID: "93c023af-9183-406d-a9ac-9a7e0ab2fb86"). InnerVolumeSpecName "kube-api-access-nzjpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:25:25.890618 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.890541    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:25.890618 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.890571    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c023af-9183-406d-a9ac-9a7e0ab2fb86-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:25.890618 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:25.890582    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjpk\" (UniqueName: \"kubernetes.io/projected/93c023af-9183-406d-a9ac-9a7e0ab2fb86-kube-api-access-nzjpk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:26.576386 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:26.576356    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"
Nov 01 01:25:26.576553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:26.576351    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d" event={"ID":"93c023af-9183-406d-a9ac-9a7e0ab2fb86","Type":"ContainerDied","Data":"d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664"}
Nov 01 01:25:26.576553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:26.576480    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b6a0e9a8506031cee143fe8e307e08466aeacb94b86ac1b3d5c5b7b1884664"
Nov 01 01:25:31.826258 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:31.826223    4493 scope.go:122] "RemoveContainer" containerID="ad4c75fd99514a128230794f51075c8026bbca7c1dbc2052486761b16f6f2912"
Nov 01 01:25:33.311665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.311624    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"]
Nov 01 01:25:33.312088 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.312034    4493 cpu_manager.go:414] "containerMap: removing container" podUID="93c023af-9183-406d-a9ac-9a7e0ab2fb86" containerName="revoke-keys"
Nov 01 01:25:33.312088 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.312045    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c023af-9183-406d-a9ac-9a7e0ab2fb86" containerName="revoke-keys"
Nov 01 01:25:33.312197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.312099    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c023af-9183-406d-a9ac-9a7e0ab2fb86" containerName="revoke-keys"
Nov 01 01:25:33.316571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.316544    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.372500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.372471    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:25:33.372770 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.372751    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:25:33.422628 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.422596    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:25:33.422628 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.422616    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:25:33.425077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.425051    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"]
Nov 01 01:25:33.458687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.458656    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzd8w\" (UniqueName: \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.458856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.458705    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.458856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.458791    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.559351 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.559317    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzd8w\" (UniqueName: \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.559528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.559419    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.559528 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.559464    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.560096 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.560074    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.562025 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.561975    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.621540 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.621510    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzd8w\" (UniqueName: \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\") pod \"maas-api-revoke-sub-5a29efb3-p5hlr\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:33.629183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:33.629160    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:34.006181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:34.003298    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"]
Nov 01 01:25:34.006880 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:25:34.006848    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdbfe80_27ba_4143_af54_cf3ca511ffe8.slice/crio-f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329 WatchSource:0}: Error finding container f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329: Status 404 returned error can't find the container with id f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329
Nov 01 01:25:34.617969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:34.617884    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr" event={"ID":"2cdbfe80-27ba-4143-af54-cf3ca511ffe8","Type":"ContainerStarted","Data":"e556c850166eb3830ee8b6e93b562c1c87049a39aad8fe6d4ee7519c6b715cd1"}
Nov 01 01:25:34.617969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:34.617935    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr" event={"ID":"2cdbfe80-27ba-4143-af54-cf3ca511ffe8","Type":"ContainerStarted","Data":"f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329"}
Nov 01 01:25:34.734018 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:34.733966    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr" podStartSLOduration=1.733950236 podStartE2EDuration="1.733950236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:25:34.733487043 +0000 UTC m=+3724.411073984" watchObservedRunningTime="2026-11-01 01:25:34.733950236 +0000 UTC m=+3724.411537203"
Nov 01 01:25:35.492308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:35.492274    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:25:35.624149 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:35.624108    4493 generic.go:356] "Generic (PLEG): container finished" podID="2cdbfe80-27ba-4143-af54-cf3ca511ffe8" containerID="e556c850166eb3830ee8b6e93b562c1c87049a39aad8fe6d4ee7519c6b715cd1" exitCode=0
Nov 01 01:25:35.624564 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:35.624194    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr" event={"ID":"2cdbfe80-27ba-4143-af54-cf3ca511ffe8","Type":"ContainerDied","Data":"e556c850166eb3830ee8b6e93b562c1c87049a39aad8fe6d4ee7519c6b715cd1"}
Nov 01 01:25:37.125159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.123349    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:37.224349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.220607    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\") pod \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") "
Nov 01 01:25:37.224349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.220691    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\" (UniqueName: \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\") pod \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") "
Nov 01 01:25:37.224349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.220807    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\") pod \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\" (UID: \"2cdbfe80-27ba-4143-af54-cf3ca511ffe8\") "
Nov 01 01:25:37.224349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.221554    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca" pod "2cdbfe80-27ba-4143-af54-cf3ca511ffe8" (UID: "2cdbfe80-27ba-4143-af54-cf3ca511ffe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:25:37.230157 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.225186    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w" pod "2cdbfe80-27ba-4143-af54-cf3ca511ffe8" (UID: "2cdbfe80-27ba-4143-af54-cf3ca511ffe8"). InnerVolumeSpecName "kube-api-access-jzd8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:25:37.265526 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.265485    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca" pod "2cdbfe80-27ba-4143-af54-cf3ca511ffe8" (UID: "2cdbfe80-27ba-4143-af54-cf3ca511ffe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:25:37.335219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.332164    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:37.335219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.332203    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzd8w\" (UniqueName: \"kubernetes.io/projected/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-kube-api-access-jzd8w\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:37.335219 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.332221    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbfe80-27ba-4143-af54-cf3ca511ffe8-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:37.657434 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.657341    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr" event={"ID":"2cdbfe80-27ba-4143-af54-cf3ca511ffe8","Type":"ContainerDied","Data":"f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329"}
Nov 01 01:25:37.677274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.659050    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"
Nov 01 01:25:37.693594 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:37.693566    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99ca907e104cc808066a512a71ffecdc0f97eb12b7545f916ac296f472fa329"
Nov 01 01:25:44.990154 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:25:44.990054    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:25:45.052562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:45.052521    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"]
Nov 01 01:25:45.102616 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:45.102583    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4tgg5"]
Nov 01 01:25:46.186997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:46.186954    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:25:46.501964 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:46.501932    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6" path="/var/lib/kubelet/pods/7af9c0ca-0f66-4529-8fd8-62f1a1f4a2f6/volumes"
Nov 01 01:25:50.542373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.542337    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"]
Nov 01 01:25:50.592231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.592201    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7mbl"]
Nov 01 01:25:50.663435 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.663399    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"]
Nov 01 01:25:50.663766 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.663754    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2cdbfe80-27ba-4143-af54-cf3ca511ffe8" containerName="revoke-keys"
Nov 01 01:25:50.663813 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.663768    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdbfe80-27ba-4143-af54-cf3ca511ffe8" containerName="revoke-keys"
Nov 01 01:25:50.663855 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.663843    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdbfe80-27ba-4143-af54-cf3ca511ffe8" containerName="revoke-keys"
Nov 01 01:25:50.668267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.668252    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.709065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.709035    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.709065 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.709070    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.709298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.709092    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdp9\" (UniqueName: \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.724063 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.724038    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:25:50.724308 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.724240    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:25:50.724430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.724404    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:25:50.724531 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.724516    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:25:50.727260 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.727237    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"]
Nov 01 01:25:50.810468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.810386    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.810468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.810422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdp9\" (UniqueName: \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.810653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.810513    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.811008 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.810987    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.812944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.812909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.871976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.871937    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdp9\" (UniqueName: \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\") pod \"maas-api-revoke-sub-3d4156f1-lbxcm\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:50.981329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:50.981290    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:51.354234 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:51.354210    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"]
Nov 01 01:25:51.355829 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:25:51.355801    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f7eae2_04d2_4153_bda2_978a20bd99ef.slice/crio-5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7 WatchSource:0}: Error finding container 5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7: Status 404 returned error can't find the container with id 5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7
Nov 01 01:25:51.732925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:51.732890    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm" event={"ID":"a1f7eae2-04d2-4153-bda2-978a20bd99ef","Type":"ContainerStarted","Data":"cc06683d8f919abe66ae767f19d98f4de8df04d5f1f3f679ecc7a35b13506dbc"}
Nov 01 01:25:51.733398 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:51.732941    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm" event={"ID":"a1f7eae2-04d2-4153-bda2-978a20bd99ef","Type":"ContainerStarted","Data":"5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7"}
Nov 01 01:25:51.850419 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:51.850367    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm" podStartSLOduration=1.850350899 podStartE2EDuration="1.850350899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:25:51.848556167 +0000 UTC m=+3741.526143113" watchObservedRunningTime="2026-11-01 01:25:51.850350899 +0000 UTC m=+3741.527937844"
Nov 01 01:25:52.501919 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:52.501888    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c00388-d572-48b8-b6c9-f8f439360f9c" path="/var/lib/kubelet/pods/d8c00388-d572-48b8-b6c9-f8f439360f9c/volumes"
Nov 01 01:25:52.739307 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:52.739269    4493 generic.go:356] "Generic (PLEG): container finished" podID="a1f7eae2-04d2-4153-bda2-978a20bd99ef" containerID="cc06683d8f919abe66ae767f19d98f4de8df04d5f1f3f679ecc7a35b13506dbc" exitCode=0
Nov 01 01:25:52.739789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:52.739359    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm" event={"ID":"a1f7eae2-04d2-4153-bda2-978a20bd99ef","Type":"ContainerDied","Data":"cc06683d8f919abe66ae767f19d98f4de8df04d5f1f3f679ecc7a35b13506dbc"}
Nov 01 01:25:53.880274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.880245    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:53.936251 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.936219    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\" (UniqueName: \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\") pod \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") "
Nov 01 01:25:53.936457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.936318    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\") pod \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") "
Nov 01 01:25:53.936457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.936347    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\") pod \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\" (UID: \"a1f7eae2-04d2-4153-bda2-978a20bd99ef\") "
Nov 01 01:25:53.936755 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.936723    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca" pod "a1f7eae2-04d2-4153-bda2-978a20bd99ef" (UID: "a1f7eae2-04d2-4153-bda2-978a20bd99ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:25:53.938608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.938586    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9" pod "a1f7eae2-04d2-4153-bda2-978a20bd99ef" (UID: "a1f7eae2-04d2-4153-bda2-978a20bd99ef"). InnerVolumeSpecName "kube-api-access-9rdp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:25:53.938692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:53.938616    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca" pod "a1f7eae2-04d2-4153-bda2-978a20bd99ef" (UID: "a1f7eae2-04d2-4153-bda2-978a20bd99ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:25:54.037693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.037618    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:54.037693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.037647    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdp9\" (UniqueName: \"kubernetes.io/projected/a1f7eae2-04d2-4153-bda2-978a20bd99ef-kube-api-access-9rdp9\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:54.037693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.037657    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7eae2-04d2-4153-bda2-978a20bd99ef-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:25:54.749761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.749726    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"
Nov 01 01:25:54.749761 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.749742    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm" event={"ID":"a1f7eae2-04d2-4153-bda2-978a20bd99ef","Type":"ContainerDied","Data":"5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7"}
Nov 01 01:25:54.749978 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:25:54.749820    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5544428fd6680e742157989636d92e888628b1edf13fd076057b52ba8929cde7"
Nov 01 01:26:07.281158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.281100    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:26:07.281669 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.281491    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1f7eae2-04d2-4153-bda2-978a20bd99ef" containerName="revoke-keys"
Nov 01 01:26:07.281669 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.281503    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f7eae2-04d2-4153-bda2-978a20bd99ef" containerName="revoke-keys"
Nov 01 01:26:07.281669 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.281578    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f7eae2-04d2-4153-bda2-978a20bd99ef" containerName="revoke-keys"
Nov 01 01:26:07.284388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.284372    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.340164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.338364    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-43c48b-openshift-default-dockercfg-kh9fz\""
Nov 01 01:26:07.366172 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366118    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366396 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366190    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366396 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366287    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366396 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366346    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366409    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366495    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366569    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.366813 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.366596    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sgk\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.396719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.396684    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:26:07.467740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467704    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467748    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467769    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467787    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sgk\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467826    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467846    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467862    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467901    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.467961 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.467958    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.468474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.468230    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.468474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.468263    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.468474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.468403    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.468474 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.468422    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.468691 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.468641    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.470739 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.470713    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.470856 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.470762    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.529660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.529628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.529840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.529712    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sgk\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\") pod \"e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") " pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:07.597418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:07.597332    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:08.025459 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.025433    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:26:08.028235 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:26:08.028200    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b0e7a3_8495_4191_8ecb_67efca39ba9f.slice/crio-9f4facf5ca4d404305639e4c14d710dd380836c5f4113a0923f3e8480a27d4d4 WatchSource:0}: Error finding container 9f4facf5ca4d404305639e4c14d710dd380836c5f4113a0923f3e8480a27d4d4: Status 404 returned error can't find the container with id 9f4facf5ca4d404305639e4c14d710dd380836c5f4113a0923f3e8480a27d4d4
Nov 01 01:26:08.030441 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.030408    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:08.030519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.030477    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:08.030519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.030505    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:08.827909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.827873    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerStarted","Data":"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"}
Nov 01 01:26:08.828335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.827934    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerStarted","Data":"9f4facf5ca4d404305639e4c14d710dd380836c5f4113a0923f3e8480a27d4d4"}
Nov 01 01:26:08.999067 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:08.999010    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podStartSLOduration=1.998996379 podStartE2EDuration="1.998996379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:26:08.997841967 +0000 UTC m=+3758.675428914" watchObservedRunningTime="2026-11-01 01:26:08.998996379 +0000 UTC m=+3758.676583323"
Nov 01 01:26:09.598553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:09.598519    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:09.603525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:09.603498    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:09.832367 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:09.832339    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:09.833349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:09.833331    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:14.989538 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:26:14.989498    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:26:18.054468 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:18.054436    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"]
Nov 01 01:26:18.102939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:18.102896    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zjqpk"]
Nov 01 01:26:18.501729 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:18.501696    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e019e836-9527-4a8e-8019-5ff75f6af161" path="/var/lib/kubelet/pods/e019e836-9527-4a8e-8019-5ff75f6af161/volumes"
Nov 01 01:26:26.308217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:26.308182    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:26:30.048971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:30.048937    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"]
Nov 01 01:26:30.098877 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:30.098841    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9clp2"]
Nov 01 01:26:30.501286 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:30.501251    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16777a7b-79b8-4c26-9384-bcdc5ae0c0e3" path="/var/lib/kubelet/pods/16777a7b-79b8-4c26-9384-bcdc5ae0c0e3/volumes"
Nov 01 01:26:31.847728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:31.847698    4493 scope.go:122] "RemoveContainer" containerID="564da360e983458886460a4c0cf4be1a146db9ea55addc2ff8bd6c6c3966f163"
Nov 01 01:26:35.988732 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.988703    4493 generic.go:356] "Generic (PLEG): container finished" podID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerID="3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233" exitCode=0
Nov 01 01:26:35.989239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.988773    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerDied","Data":"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"}
Nov 01 01:26:35.989239 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.989189    4493 scope.go:122] "RemoveContainer" containerID="3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"
Nov 01 01:26:35.990351 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.990319    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:35.990416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.990396    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:35.990457 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:35.990429    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:26:36.997574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:36.997543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerStarted","Data":"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"}
Nov 01 01:26:37.598299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:37.598258    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:37.599524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:37.599498    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 01:26:37.599659 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:37.599548    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 01:26:38.598017 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:38.597982    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 01:26:38.598458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:38.598038    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 01:26:39.597931 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:39.597896    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 01:26:39.598116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:39.597954    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 01:26:39.834158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:39.833945    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:40.598613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:40.598574    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 01:26:40.599104 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:40.598650    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 01:26:41.598066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:41.598028    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 01:26:41.598283 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:41.598090    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 01:26:42.602738 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:42.602711    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:42.603810 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:42.603791    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:26:44.173978 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:26:44.173926    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:26:44.990277 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:26:44.990241    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:27:09.047982 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:09.047940    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"]
Nov 01 01:27:09.097726 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:09.097695    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mlcn5"]
Nov 01 01:27:10.501607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:10.501574    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294f5e67-53c2-4f51-b519-4fdcecb5e196" path="/var/lib/kubelet/pods/294f5e67-53c2-4f51-b519-4fdcecb5e196/volumes"
Nov 01 01:27:10.727083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:10.727050    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:27:14.989446 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:27:14.989367    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:27:20.048098 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:20.048063    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"]
Nov 01 01:27:20.098159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:20.098115    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xfrb7"]
Nov 01 01:27:20.500717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:20.500677    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c41a569-ca9c-489c-8652-5a665589f13b" path="/var/lib/kubelet/pods/6c41a569-ca9c-489c-8652-5a665589f13b/volumes"
Nov 01 01:27:25.011175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.011124    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"]
Nov 01 01:27:25.015802 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.015784    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.071403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.071370    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:27:25.071403 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.071390    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:27:25.121208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.121174    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:27:25.121392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.121247    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:27:25.123366 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.123347    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"]
Nov 01 01:27:25.127707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.127685    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.127773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.127733    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.127898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.127885    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfdn\" (UniqueName: \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.228903 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.228868    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.229080 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.228989    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfdn\" (UniqueName: \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.229080 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.229035    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.229790 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.229762    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.231551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.231524    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.290994 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.290920    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfdn\" (UniqueName: \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\") pod \"maas-api-revoke-sub-9f086539-4ww68\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.328867 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.328834    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:25.703561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:25.703420    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"]
Nov 01 01:27:25.705769 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:27:25.705742    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3430fc1c_8fc8_46b6_89a4_4a20358086b5.slice/crio-9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e WatchSource:0}: Error finding container 9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e: Status 404 returned error can't find the container with id 9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e
Nov 01 01:27:26.256305 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:26.256269    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68" event={"ID":"3430fc1c-8fc8-46b6-89a4-4a20358086b5","Type":"ContainerStarted","Data":"09ad9770c05729e61da985cf5f36110e77a65e8678feead7e343df0021780f1c"}
Nov 01 01:27:26.256713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:26.256329    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68" event={"ID":"3430fc1c-8fc8-46b6-89a4-4a20358086b5","Type":"ContainerStarted","Data":"9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e"}
Nov 01 01:27:26.372994 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:26.372889    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68" podStartSLOduration=2.372875916 podStartE2EDuration="2.372875916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:27:26.372305113 +0000 UTC m=+3836.049892059" watchObservedRunningTime="2026-11-01 01:27:26.372875916 +0000 UTC m=+3836.050462860"
Nov 01 01:27:27.262871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:27.262834    4493 generic.go:356] "Generic (PLEG): container finished" podID="3430fc1c-8fc8-46b6-89a4-4a20358086b5" containerID="09ad9770c05729e61da985cf5f36110e77a65e8678feead7e343df0021780f1c" exitCode=0
Nov 01 01:27:27.263350 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:27.262924    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68" event={"ID":"3430fc1c-8fc8-46b6-89a4-4a20358086b5","Type":"ContainerDied","Data":"09ad9770c05729e61da985cf5f36110e77a65e8678feead7e343df0021780f1c"}
Nov 01 01:27:28.400057 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.400031    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:28.465960 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.465926    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\") pod \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") "
Nov 01 01:27:28.465960 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.465965    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\" (UniqueName: \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\") pod \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") "
Nov 01 01:27:28.466223 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.466011    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\") pod \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\" (UID: \"3430fc1c-8fc8-46b6-89a4-4a20358086b5\") "
Nov 01 01:27:28.466455 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.466431    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca" pod "3430fc1c-8fc8-46b6-89a4-4a20358086b5" (UID: "3430fc1c-8fc8-46b6-89a4-4a20358086b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:27:28.468230 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.468204    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca" pod "3430fc1c-8fc8-46b6-89a4-4a20358086b5" (UID: "3430fc1c-8fc8-46b6-89a4-4a20358086b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:27:28.468365 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.468288    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn" pod "3430fc1c-8fc8-46b6-89a4-4a20358086b5" (UID: "3430fc1c-8fc8-46b6-89a4-4a20358086b5"). InnerVolumeSpecName "kube-api-access-tsfdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:27:28.567316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.567237    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:28.567316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.567267    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3430fc1c-8fc8-46b6-89a4-4a20358086b5-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:28.567316 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:28.567276    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfdn\" (UniqueName: \"kubernetes.io/projected/3430fc1c-8fc8-46b6-89a4-4a20358086b5-kube-api-access-tsfdn\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:29.275480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:29.275450    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"
Nov 01 01:27:29.275480 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:29.275475    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68" event={"ID":"3430fc1c-8fc8-46b6-89a4-4a20358086b5","Type":"ContainerDied","Data":"9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e"}
Nov 01 01:27:29.275699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:29.275541    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e24d38afb1a220a87bdc6c324c8d2b2ecfab6904ae1337361b02128a2531c7e"
Nov 01 01:27:31.870579 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:31.870549    4493 scope.go:122] "RemoveContainer" containerID="4b57c1610d0bbed8a765e77a692c6b283f1b65cb84293b037069a0126b9cca5c"
Nov 01 01:27:31.879547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:31.879524    4493 scope.go:122] "RemoveContainer" containerID="cdc4feacb467f8f5942cbd75a939100ec952192a8d262904dde861acb50e5c8c"
Nov 01 01:27:31.887275 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:31.887243    4493 scope.go:122] "RemoveContainer" containerID="566dc9efbdc63a03533a4aca88fb2fdc021748c841544d643e3150dc0dde063e"
Nov 01 01:27:37.231591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.231552    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"]
Nov 01 01:27:37.232000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.231916    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3430fc1c-8fc8-46b6-89a4-4a20358086b5" containerName="revoke-keys"
Nov 01 01:27:37.232000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.231926    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3430fc1c-8fc8-46b6-89a4-4a20358086b5" containerName="revoke-keys"
Nov 01 01:27:37.232079 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.232015    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3430fc1c-8fc8-46b6-89a4-4a20358086b5" containerName="revoke-keys"
Nov 01 01:27:37.235117 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.235101    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.287416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.287382    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:27:37.337365 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.337335    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:27:37.337542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.337371    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:27:37.337542 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.337399    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:27:37.339623 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.339604    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"]
Nov 01 01:27:37.358517 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.358490    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.358644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.358535    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.358687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.358646    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpgb\" (UniqueName: \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.460179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.460150    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpgb\" (UniqueName: \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.460342 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.460250    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.460342 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.460290    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.461023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.461004    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.462768 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.462751    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.521189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.521102    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpgb\" (UniqueName: \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.548000 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.547971    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:37.919699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:37.919674    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"]
Nov 01 01:27:37.921618 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:27:37.921594    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3028b496_b342_4de9_b6f1_29bcb34d285a.slice/crio-16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5 WatchSource:0}: Error finding container 16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5: Status 404 returned error can't find the container with id 16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5
Nov 01 01:27:38.319481 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:38.319446    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt" event={"ID":"3028b496-b342-4de9-b6f1-29bcb34d285a","Type":"ContainerStarted","Data":"7a291eea611804335e0eccf05dff0102d3abcd1e74c3a1ae392649b24311f780"}
Nov 01 01:27:38.319974 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:38.319512    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt" event={"ID":"3028b496-b342-4de9-b6f1-29bcb34d285a","Type":"ContainerStarted","Data":"16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5"}
Nov 01 01:27:38.436236 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:38.436179    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt" podStartSLOduration=1.436164206 podStartE2EDuration="1.436164206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:27:38.433996053 +0000 UTC m=+3848.111582997" watchObservedRunningTime="2026-11-01 01:27:38.436164206 +0000 UTC m=+3848.113751150"
Nov 01 01:27:39.326364 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:39.326326    4493 generic.go:356] "Generic (PLEG): container finished" podID="3028b496-b342-4de9-b6f1-29bcb34d285a" containerID="7a291eea611804335e0eccf05dff0102d3abcd1e74c3a1ae392649b24311f780" exitCode=0
Nov 01 01:27:39.326808 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:39.326394    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt" event={"ID":"3028b496-b342-4de9-b6f1-29bcb34d285a","Type":"ContainerDied","Data":"7a291eea611804335e0eccf05dff0102d3abcd1e74c3a1ae392649b24311f780"}
Nov 01 01:27:40.006728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.006681    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:27:40.470822 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.470793    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:40.602507 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.602419    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\") pod \"3028b496-b342-4de9-b6f1-29bcb34d285a\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") "
Nov 01 01:27:40.602843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.602808    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\") pod \"3028b496-b342-4de9-b6f1-29bcb34d285a\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") "
Nov 01 01:27:40.602986 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.602886    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\" (UniqueName: \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\") pod \"3028b496-b342-4de9-b6f1-29bcb34d285a\" (UID: \"3028b496-b342-4de9-b6f1-29bcb34d285a\") "
Nov 01 01:27:40.603766 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.603727    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca" pod "3028b496-b342-4de9-b6f1-29bcb34d285a" (UID: "3028b496-b342-4de9-b6f1-29bcb34d285a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:27:40.618650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.618617    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca" pod "3028b496-b342-4de9-b6f1-29bcb34d285a" (UID: "3028b496-b342-4de9-b6f1-29bcb34d285a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:27:40.640438 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.640404    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb" pod "3028b496-b342-4de9-b6f1-29bcb34d285a" (UID: "3028b496-b342-4de9-b6f1-29bcb34d285a"). InnerVolumeSpecName "kube-api-access-2cpgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:27:40.711173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.707986    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:40.711173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.708020    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cpgb\" (UniqueName: \"kubernetes.io/projected/3028b496-b342-4de9-b6f1-29bcb34d285a-kube-api-access-2cpgb\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:40.711173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:40.708039    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3028b496-b342-4de9-b6f1-29bcb34d285a-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:27:41.338390 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:41.338355    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"
Nov 01 01:27:41.338563 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:41.338391    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt" event={"ID":"3028b496-b342-4de9-b6f1-29bcb34d285a","Type":"ContainerDied","Data":"16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5"}
Nov 01 01:27:41.338563 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:41.338441    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16dc41fa9493e7bff41290f01d153e116388ed14bbcfb9334092b9a44fc20aa5"
Nov 01 01:27:44.988762 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:27:44.988726    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:27:57.222208 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:57.222169    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"]
Nov 01 01:27:57.271777 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:57.271744    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-43c48b-vxpxt"]
Nov 01 01:27:58.047645 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:58.047607    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"]
Nov 01 01:27:58.097489 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:58.097452    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xs4ph"]
Nov 01 01:27:58.501005 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:58.500974    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3028b496-b342-4de9-b6f1-29bcb34d285a" path="/var/lib/kubelet/pods/3028b496-b342-4de9-b6f1-29bcb34d285a/volumes"
Nov 01 01:27:58.501424 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:58.501409    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d9abba-db0c-471a-b69f-e6d6c4d2bfdf" path="/var/lib/kubelet/pods/77d9abba-db0c-471a-b69f-e6d6c4d2bfdf/volumes"
Nov 01 01:27:59.770378 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:27:59.770345    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:28:04.109733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:04.109700    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:28:04.110173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:04.109955    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy" containerID="cri-o://534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4" gracePeriod=30
Nov 01 01:28:05.772788 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.772749    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"]
Nov 01 01:28:05.822506 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.822471    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lbxcm"]
Nov 01 01:28:05.898463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.898431    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"]
Nov 01 01:28:05.898790 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.898779    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3028b496-b342-4de9-b6f1-29bcb34d285a" containerName="revoke-keys"
Nov 01 01:28:05.898836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.898791    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3028b496-b342-4de9-b6f1-29bcb34d285a" containerName="revoke-keys"
Nov 01 01:28:05.898887 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.898878    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3028b496-b342-4de9-b6f1-29bcb34d285a" containerName="revoke-keys"
Nov 01 01:28:05.903218 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.903203    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:05.958572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.958524    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:28:05.958767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.958628    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:28:05.958842 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.958769    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:28:05.958947 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.958928    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:28:05.961216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.961191    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"]
Nov 01 01:28:05.965167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.965144    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:05.965327 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.965308    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:05.965412 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:05.965343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l65\" (UniqueName: \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.066423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.066334    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.066423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.066406    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.066423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.066426    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l65\" (UniqueName: \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.067111 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.067086    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.068975 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.068956    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.128377 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.128347    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l65\" (UniqueName: \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\") pod \"maas-api-revoke-sub-3d4156f1-265dj\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.216575 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.216541    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:06.501749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.501717    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f7eae2-04d2-4153-bda2-978a20bd99ef" path="/var/lib/kubelet/pods/a1f7eae2-04d2-4153-bda2-978a20bd99ef/volumes"
Nov 01 01:28:06.593622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:06.593584    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"]
Nov 01 01:28:06.595092 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:28:06.595068    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552130aa_f024_499e_8d4a_968f96ad6964.slice/crio-0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b WatchSource:0}: Error finding container 0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b: Status 404 returned error can't find the container with id 0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b
Nov 01 01:28:07.467648 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:07.467613    4493 generic.go:356] "Generic (PLEG): container finished" podID="552130aa-f024-499e-8d4a-968f96ad6964" containerID="89a5135be7dd0af7acee43f49bfb9dbe7edb9574ad8717fd171051bfe244b97f" exitCode=0
Nov 01 01:28:07.468056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:07.467675    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj" event={"ID":"552130aa-f024-499e-8d4a-968f96ad6964","Type":"ContainerDied","Data":"89a5135be7dd0af7acee43f49bfb9dbe7edb9574ad8717fd171051bfe244b97f"}
Nov 01 01:28:07.468056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:07.467766    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj" event={"ID":"552130aa-f024-499e-8d4a-968f96ad6964","Type":"ContainerStarted","Data":"0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b"}
Nov 01 01:28:08.610853 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.610827    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:08.692939 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.692911    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\") pod \"552130aa-f024-499e-8d4a-968f96ad6964\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") "
Nov 01 01:28:08.693127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.692975    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\") pod \"552130aa-f024-499e-8d4a-968f96ad6964\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") "
Nov 01 01:28:08.693127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.693021    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\" (UniqueName: \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\") pod \"552130aa-f024-499e-8d4a-968f96ad6964\" (UID: \"552130aa-f024-499e-8d4a-968f96ad6964\") "
Nov 01 01:28:08.693371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.693344    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca" pod "552130aa-f024-499e-8d4a-968f96ad6964" (UID: "552130aa-f024-499e-8d4a-968f96ad6964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:28:08.695373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.695347    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65" pod "552130aa-f024-499e-8d4a-968f96ad6964" (UID: "552130aa-f024-499e-8d4a-968f96ad6964"). InnerVolumeSpecName "kube-api-access-x5l65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:28:08.695373 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.695351    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca" pod "552130aa-f024-499e-8d4a-968f96ad6964" (UID: "552130aa-f024-499e-8d4a-968f96ad6964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:28:08.794651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.794574    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:08.794651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.794604    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/552130aa-f024-499e-8d4a-968f96ad6964-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:08.794651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:08.794614    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5l65\" (UniqueName: \"kubernetes.io/projected/552130aa-f024-499e-8d4a-968f96ad6964-kube-api-access-x5l65\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.349379 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.349349    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:28:09.400731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400651    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.400888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400745    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.400888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400771    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.400888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400827    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.400888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400857    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.401109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400891    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.401109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.400974    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.401109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401006    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.401109 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401036    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\") pod \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\" (UID: \"b8b0e7a3-8495-4191-8ecb-67efca39ba9f\") "
Nov 01 01:28:09.401411 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401348    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:28:09.401411 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401384    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:28:09.401539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401491    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.401539 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401513    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.401814 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401770    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:28:09.401947 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401786    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:28:09.401947 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.401838    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:28:09.403151 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.403117    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:28:09.403276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.403252    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:28:09.403619 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.403602    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:28:09.403685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.403626    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk" pod "b8b0e7a3-8495-4191-8ecb-67efca39ba9f" (UID: "b8b0e7a3-8495-4191-8ecb-67efca39ba9f"). InnerVolumeSpecName "kube-api-access-m8sgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:28:09.481597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.481563    4493 generic.go:356] "Generic (PLEG): container finished" podID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerID="534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4" exitCode=0
Nov 01 01:28:09.481791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.481623    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"
Nov 01 01:28:09.481791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.481649    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerDied","Data":"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"}
Nov 01 01:28:09.481791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.481706    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc" event={"ID":"b8b0e7a3-8495-4191-8ecb-67efca39ba9f","Type":"ContainerDied","Data":"9f4facf5ca4d404305639e4c14d710dd380836c5f4113a0923f3e8480a27d4d4"}
Nov 01 01:28:09.481791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.481735    4493 scope.go:122] "RemoveContainer" containerID="534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"
Nov 01 01:28:09.483277 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.483253    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"
Nov 01 01:28:09.483394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.483246    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj" event={"ID":"552130aa-f024-499e-8d4a-968f96ad6964","Type":"ContainerDied","Data":"0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b"}
Nov 01 01:28:09.483394 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.483383    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfbf071dcb2567c99b5f6a369893728285655170e3b58b5a0b594296b96d95b"
Nov 01 01:28:09.492336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.492319    4493 scope.go:122] "RemoveContainer" containerID="3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"
Nov 01 01:28:09.500599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.500578    4493 scope.go:122] "RemoveContainer" containerID="534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"
Nov 01 01:28:09.500849 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:28:09.500827    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4\": container with ID starting with 534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4 not found: ID does not exist" containerID="534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"
Nov 01 01:28:09.500900 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.500858    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4"} err="failed to get container status \"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4\": rpc error: code = NotFound desc = could not find container \"534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4\": container with ID starting with 534f37daa63101a4e83f50dde9e45b5a5aed09882198e25fc883d410ccd5e4c4 not found: ID does not exist"
Nov 01 01:28:09.500900 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.500875    4493 scope.go:122] "RemoveContainer" containerID="3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"
Nov 01 01:28:09.501119 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:28:09.501103    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233\": container with ID starting with 3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233 not found: ID does not exist" containerID="3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"
Nov 01 01:28:09.501197 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.501124    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233"} err="failed to get container status \"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233\": rpc error: code = NotFound desc = could not find container \"3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233\": container with ID starting with 3ac7eaa3c77cae2cbf1a968bb155cd37e065aadb34ada7318949433bb14c7233 not found: ID does not exist"
Nov 01 01:28:09.502559 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502541    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502562    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502571    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502580    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502588    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502597    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.502607 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.502606    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8sgk\" (UniqueName: \"kubernetes.io/projected/b8b0e7a3-8495-4191-8ecb-67efca39ba9f-kube-api-access-m8sgk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:28:09.688246 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.688158    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:28:09.693917 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:09.693889    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-43c48b-openshift-default-c6b5c749-5d4qc"]
Nov 01 01:28:10.501463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:10.501426    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" path="/var/lib/kubelet/pods/b8b0e7a3-8495-4191-8ecb-67efca39ba9f/volumes"
Nov 01 01:28:14.988640 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:28:14.988604    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:28:26.047301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:26.047265    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"]
Nov 01 01:28:26.096678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:26.096648    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dj6j5"]
Nov 01 01:28:26.501416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:26.501384    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418e7eab-301e-4bdd-8dc9-d1bc26055629" path="/var/lib/kubelet/pods/418e7eab-301e-4bdd-8dc9-d1bc26055629/volumes"
Nov 01 01:28:31.913825 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:31.913791    4493 scope.go:122] "RemoveContainer" containerID="0c41a514eaf2675eb30e6cb7d2e2d4c0b16867e8ae3e00a4a8c2e1d4e0d53c69"
Nov 01 01:28:31.922278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:31.922256    4493 scope.go:122] "RemoveContainer" containerID="98aac924359e089fdcead6378da47970c487abd2861e7c87ba3a262358ff051a"
Nov 01 01:28:44.989155 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:28:44.989060    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:28:58.050253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:58.050217    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"]
Nov 01 01:28:58.099954 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:58.099925    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qn8h6"]
Nov 01 01:28:58.500980 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:28:58.500949    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668470cd-74dd-4bcf-92f3-db9bb4754fc9" path="/var/lib/kubelet/pods/668470cd-74dd-4bcf-92f3-db9bb4754fc9/volumes"
Nov 01 01:29:14.988957 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:29:14.988919    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:29:19.059014 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:19.058978    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"]
Nov 01 01:29:19.111384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:19.111329    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5fp49"]
Nov 01 01:29:20.501271 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:20.501241    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfed82e-a73f-40dc-bec4-c5124aad95f7" path="/var/lib/kubelet/pods/7dfed82e-a73f-40dc-bec4-c5124aad95f7/volumes"
Nov 01 01:29:31.953788 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:31.953752    4493 scope.go:122] "RemoveContainer" containerID="eeace95fe80c463039aa5e3dc7dd979e018cc891a8167c1393145c4803e539a5"
Nov 01 01:29:31.962678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:31.962660    4493 scope.go:122] "RemoveContainer" containerID="84e2e35b1cb43f26380d9ed27ddbabe031a4af4205f7ce9d249a5603f8828694"
Nov 01 01:29:44.989585 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:29:44.989552    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:29:50.262187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:50.262150    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:29:54.838547 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838513    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"]
Nov 01 01:29:54.838934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838866    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:29:54.838934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838878    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:29:54.838934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838887    4493 cpu_manager.go:414] "containerMap: removing container" podUID="552130aa-f024-499e-8d4a-968f96ad6964" containerName="revoke-keys"
Nov 01 01:29:54.838934 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838892    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552130aa-f024-499e-8d4a-968f96ad6964" containerName="revoke-keys"
Nov 01 01:29:54.839072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838975    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:29:54.839072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838983    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="552130aa-f024-499e-8d4a-968f96ad6964" containerName="revoke-keys"
Nov 01 01:29:54.839072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.838990    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:29:54.842050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.842033    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:54.894599 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.894574    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:29:54.921276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.921241    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:54.921276 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.921280    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhd9m\" (UniqueName: \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:54.921492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.921392    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:54.944544 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.944514    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:29:54.944666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.944571    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:29:54.944666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.944641    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:29:54.946569 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:54.946544    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"]
Nov 01 01:29:55.021990 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.021961    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.022278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.022001    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhd9m\" (UniqueName: \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.022278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.022083    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.022720 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.022697    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.025232 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.025210    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.083189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.083157    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhd9m\" (UniqueName: \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\") pod \"maas-api-revoke-sub-d576e68b-xcmhn\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.155034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.154954    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:55.532126 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:55.532098    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"]
Nov 01 01:29:55.533038 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:29:55.533016    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204bef77_ee0c_45e3_b256_d6b1de1af7dd.slice/crio-43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01 WatchSource:0}: Error finding container 43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01: Status 404 returned error can't find the container with id 43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01
Nov 01 01:29:56.033693 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:56.033663    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn" event={"ID":"204bef77-ee0c-45e3-b256-d6b1de1af7dd","Type":"ContainerStarted","Data":"eb43a144daa08a27a02b9b5657a0661fd1311d0d0e31c2c454e2e9da94aed4fe"}
Nov 01 01:29:56.034058 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:56.033717    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn" event={"ID":"204bef77-ee0c-45e3-b256-d6b1de1af7dd","Type":"ContainerStarted","Data":"43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01"}
Nov 01 01:29:56.151066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:56.150952    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn" podStartSLOduration=2.150934289 podStartE2EDuration="2.150934289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:29:56.149852169 +0000 UTC m=+3985.827439113" watchObservedRunningTime="2026-11-01 01:29:56.150934289 +0000 UTC m=+3985.828521236"
Nov 01 01:29:57.039428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:57.039395    4493 generic.go:356] "Generic (PLEG): container finished" podID="204bef77-ee0c-45e3-b256-d6b1de1af7dd" containerID="eb43a144daa08a27a02b9b5657a0661fd1311d0d0e31c2c454e2e9da94aed4fe" exitCode=0
Nov 01 01:29:57.039835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:57.039472    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn" event={"ID":"204bef77-ee0c-45e3-b256-d6b1de1af7dd","Type":"ContainerDied","Data":"eb43a144daa08a27a02b9b5657a0661fd1311d0d0e31c2c454e2e9da94aed4fe"}
Nov 01 01:29:58.176427 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.176403    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:29:58.254858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.254828    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\" (UniqueName: \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\") pod \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") "
Nov 01 01:29:58.255055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.254870    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\") pod \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") "
Nov 01 01:29:58.255055 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.254925    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\") pod \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\" (UID: \"204bef77-ee0c-45e3-b256-d6b1de1af7dd\") "
Nov 01 01:29:58.255398 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.255374    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca" pod "204bef77-ee0c-45e3-b256-d6b1de1af7dd" (UID: "204bef77-ee0c-45e3-b256-d6b1de1af7dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:29:58.257138 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.257114    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m" pod "204bef77-ee0c-45e3-b256-d6b1de1af7dd" (UID: "204bef77-ee0c-45e3-b256-d6b1de1af7dd"). InnerVolumeSpecName "kube-api-access-rhd9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:29:58.257252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.257181    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca" pod "204bef77-ee0c-45e3-b256-d6b1de1af7dd" (UID: "204bef77-ee0c-45e3-b256-d6b1de1af7dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:29:58.356548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.356448    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhd9m\" (UniqueName: \"kubernetes.io/projected/204bef77-ee0c-45e3-b256-d6b1de1af7dd-kube-api-access-rhd9m\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:29:58.356548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.356495    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:29:58.356548 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:58.356508    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204bef77-ee0c-45e3-b256-d6b1de1af7dd-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:29:59.050237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:59.050202    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn" event={"ID":"204bef77-ee0c-45e3-b256-d6b1de1af7dd","Type":"ContainerDied","Data":"43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01"}
Nov 01 01:29:59.050419 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:59.050250    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43715ac6ade3bb340716fa4ebdfebf479f8b73ac90a71c7f1bd8c7adf78caf01"
Nov 01 01:29:59.050419 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:29:59.050255    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"
Nov 01 01:30:00.157441 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157406    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:00.157985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157766    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:30:00.157985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157777    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b0e7a3-8495-4191-8ecb-67efca39ba9f" containerName="istio-proxy"
Nov 01 01:30:00.157985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157800    4493 cpu_manager.go:414] "containerMap: removing container" podUID="204bef77-ee0c-45e3-b256-d6b1de1af7dd" containerName="revoke-keys"
Nov 01 01:30:00.157985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157805    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204bef77-ee0c-45e3-b256-d6b1de1af7dd" containerName="revoke-keys"
Nov 01 01:30:00.157985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.157872    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="204bef77-ee0c-45e3-b256-d6b1de1af7dd" containerName="revoke-keys"
Nov 01 01:30:00.160890 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.160872    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.213676 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.213621    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:30:00.263678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.263647    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:30:00.263678 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.263661    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:30:00.265893 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.265867    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:00.274933 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.274876    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.275077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.275025    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftq8w\" (UniqueName: \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.376066 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.376034    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.376321 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.376093    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftq8w\" (UniqueName: \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.376775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.376746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.438083 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.438012    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftq8w\" (UniqueName: \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\") pod \"maas-api-key-cleanup-29891610-4hx26\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.473895 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.473863    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:00.846780 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:00.846757    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:00.849297 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:30:00.849266    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d8aba4_378b_434b_8d83_1d61892e80ae.slice/crio-c3c6c4d4eb2d6e4ea03a42d772cca01fb2d8d4bcbefca3eb39b01bf6ab1c882b WatchSource:0}: Error finding container c3c6c4d4eb2d6e4ea03a42d772cca01fb2d8d4bcbefca3eb39b01bf6ab1c882b: Status 404 returned error can't find the container with id c3c6c4d4eb2d6e4ea03a42d772cca01fb2d8d4bcbefca3eb39b01bf6ab1c882b
Nov 01 01:30:01.061559 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:01.061523    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerStarted","Data":"c3c6c4d4eb2d6e4ea03a42d772cca01fb2d8d4bcbefca3eb39b01bf6ab1c882b"}
Nov 01 01:30:02.068525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:02.068490    4493 generic.go:356] "Generic (PLEG): container finished" podID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerID="bd06867f392a5e5053c14adb3041e9151dc2b8473fd8a9dd8635a47988c5d047" exitCode=60
Nov 01 01:30:02.069032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:02.068550    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerDied","Data":"bd06867f392a5e5053c14adb3041e9151dc2b8473fd8a9dd8635a47988c5d047"}
Nov 01 01:30:02.069032 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:02.068847    4493 scope.go:122] "RemoveContainer" containerID="bd06867f392a5e5053c14adb3041e9151dc2b8473fd8a9dd8635a47988c5d047"
Nov 01 01:30:03.075922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:03.075841    4493 generic.go:356] "Generic (PLEG): container finished" podID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerID="ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e" exitCode=60
Nov 01 01:30:03.075922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:03.075879    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerDied","Data":"ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e"}
Nov 01 01:30:03.075922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:03.075921    4493 scope.go:122] "RemoveContainer" containerID="bd06867f392a5e5053c14adb3041e9151dc2b8473fd8a9dd8635a47988c5d047"
Nov 01 01:30:03.076612 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:03.076203    4493 scope.go:122] "RemoveContainer" containerID="ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e"
Nov 01 01:30:03.076612 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:30:03.076454    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891610-4hx26_odh-ai-gateway-infra(77d8aba4-378b-434b-8d83-1d61892e80ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae"
Nov 01 01:30:04.087257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:04.087227    4493 scope.go:122] "RemoveContainer" containerID="ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e"
Nov 01 01:30:04.087734 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:30:04.087562    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891610-4hx26_odh-ai-gateway-infra(77d8aba4-378b-434b-8d83-1d61892e80ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae"
Nov 01 01:30:07.668463 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:07.668426    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:30:13.496589 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:13.496487    4493 scope.go:122] "RemoveContainer" containerID="ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e"
Nov 01 01:30:14.173215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:14.172654    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerStarted","Data":"dfe90242cc84aaa1b1f9831690132fe284d4eef84adb55a120c63bd728b44165"}
Nov 01 01:30:14.308159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:14.292050    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" podStartSLOduration=14.292032123 podStartE2EDuration="14.292032123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:14.290533883 +0000 UTC m=+4003.968120831" watchObservedRunningTime="2026-11-01 01:30:14.292032123 +0000 UTC m=+4003.969619070"
Nov 01 01:30:15.114433 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:30:15.114335    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:30:15.212158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:15.211676    4493 generic.go:356] "Generic (PLEG): container finished" podID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerID="dfe90242cc84aaa1b1f9831690132fe284d4eef84adb55a120c63bd728b44165" exitCode=60
Nov 01 01:30:15.212158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:15.211868    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerDied","Data":"dfe90242cc84aaa1b1f9831690132fe284d4eef84adb55a120c63bd728b44165"}
Nov 01 01:30:15.212158 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:15.211927    4493 scope.go:122] "RemoveContainer" containerID="ef9245adf4af577176070c1e540c58f4d5f3a30d6a6b09e5a68f96587013c39e"
Nov 01 01:30:15.212500 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:15.212410    4493 scope.go:122] "RemoveContainer" containerID="dfe90242cc84aaa1b1f9831690132fe284d4eef84adb55a120c63bd728b44165"
Nov 01 01:30:15.216162 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:30:15.212710    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891610-4hx26_odh-ai-gateway-infra(77d8aba4-378b-434b-8d83-1d61892e80ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae"
Nov 01 01:30:15.324418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:15.324316    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:16.351400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.351360    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:16.505880 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.505794    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\") pod \"77d8aba4-378b-434b-8d83-1d61892e80ae\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") "
Nov 01 01:30:16.506024 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.505951    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\" (UniqueName: \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\") pod \"77d8aba4-378b-434b-8d83-1d61892e80ae\" (UID: \"77d8aba4-378b-434b-8d83-1d61892e80ae\") "
Nov 01 01:30:16.506193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.506170    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca" pod "77d8aba4-378b-434b-8d83-1d61892e80ae" (UID: "77d8aba4-378b-434b-8d83-1d61892e80ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:30:16.506306 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.506291    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d8aba4-378b-434b-8d83-1d61892e80ae-maas-api-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:30:16.508204 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.508173    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w" pod "77d8aba4-378b-434b-8d83-1d61892e80ae" (UID: "77d8aba4-378b-434b-8d83-1d61892e80ae"). InnerVolumeSpecName "kube-api-access-ftq8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:30:16.606994 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:16.606964    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftq8w\" (UniqueName: \"kubernetes.io/projected/77d8aba4-378b-434b-8d83-1d61892e80ae-kube-api-access-ftq8w\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:30:17.175782 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.175738    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:30:17.223772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.223741    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26" event={"ID":"77d8aba4-378b-434b-8d83-1d61892e80ae","Type":"ContainerDied","Data":"c3c6c4d4eb2d6e4ea03a42d772cca01fb2d8d4bcbefca3eb39b01bf6ab1c882b"}
Nov 01 01:30:17.223772 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.223759    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"
Nov 01 01:30:17.224078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.223805    4493 scope.go:122] "RemoveContainer" containerID="dfe90242cc84aaa1b1f9831690132fe284d4eef84adb55a120c63bd728b44165"
Nov 01 01:30:17.388289 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.388238    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:17.437820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:17.437740    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891610-4hx26"]
Nov 01 01:30:18.501634 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:18.501603    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" path="/var/lib/kubelet/pods/77d8aba4-378b-434b-8d83-1d61892e80ae/volumes"
Nov 01 01:30:23.915561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.915530    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"]
Nov 01 01:30:23.916107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916062    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.916107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916081    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.916107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916100    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.916107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916108    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.916377 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916247    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.916377 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.916273    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:23.920492 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.920471    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:23.963642 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.963614    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:23.963797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.963672    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58nw\" (UniqueName: \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:23.963797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:23.963690    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.025943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.025913    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:30:24.025943 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.025923    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:30:24.026211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.025962    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:30:24.026211 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.026016    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:30:24.028056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.028037    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"]
Nov 01 01:30:24.064658 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.064631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58nw\" (UniqueName: \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.064767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.064661    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.064767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.064748    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.065388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.065371    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.067302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.067286    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.126038 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.126001    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58nw\" (UniqueName: \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\") pod \"maas-api-revoke-sub-d2bc93bc-c5cfc\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.232920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.232881    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:24.606715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:24.606632    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"]
Nov 01 01:30:24.609325 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:30:24.609294    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c604ae_5b99_40d4_998a_d5e2a18564e7.slice/crio-2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f WatchSource:0}: Error finding container 2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f: Status 404 returned error can't find the container with id 2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f
Nov 01 01:30:25.266110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:25.266073    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc" event={"ID":"e0c604ae-5b99-40d4-998a-d5e2a18564e7","Type":"ContainerStarted","Data":"0a8f8a3dfe2d6ad1b4fbe887cf7019391b641fd05ba9a253eca762737ad28870"}
Nov 01 01:30:25.266508 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:25.266150    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc" event={"ID":"e0c604ae-5b99-40d4-998a-d5e2a18564e7","Type":"ContainerStarted","Data":"2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f"}
Nov 01 01:30:25.382073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:25.382024    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc" podStartSLOduration=2.382008756 podStartE2EDuration="2.382008756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:25.381979744 +0000 UTC m=+4015.059566691" watchObservedRunningTime="2026-11-01 01:30:25.382008756 +0000 UTC m=+4015.059595702"
Nov 01 01:30:26.046927 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:26.046889    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"]
Nov 01 01:30:26.049840 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:26.049813    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fm25d"]
Nov 01 01:30:26.272477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:26.272448    4493 generic.go:356] "Generic (PLEG): container finished" podID="e0c604ae-5b99-40d4-998a-d5e2a18564e7" containerID="0a8f8a3dfe2d6ad1b4fbe887cf7019391b641fd05ba9a253eca762737ad28870" exitCode=0
Nov 01 01:30:26.272923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:26.272485    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc" event={"ID":"e0c604ae-5b99-40d4-998a-d5e2a18564e7","Type":"ContainerDied","Data":"0a8f8a3dfe2d6ad1b4fbe887cf7019391b641fd05ba9a253eca762737ad28870"}
Nov 01 01:30:26.501205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:26.501176    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c023af-9183-406d-a9ac-9a7e0ab2fb86" path="/var/lib/kubelet/pods/93c023af-9183-406d-a9ac-9a7e0ab2fb86/volumes"
Nov 01 01:30:27.417571 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.417548    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:27.495811 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.495784    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\") pod \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") "
Nov 01 01:30:27.495987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.495890    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\" (UniqueName: \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\") pod \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") "
Nov 01 01:30:27.495987 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.495933    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\") pod \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\" (UID: \"e0c604ae-5b99-40d4-998a-d5e2a18564e7\") "
Nov 01 01:30:27.496327 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.496300    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca" pod "e0c604ae-5b99-40d4-998a-d5e2a18564e7" (UID: "e0c604ae-5b99-40d4-998a-d5e2a18564e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:30:27.498127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.498105    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca" pod "e0c604ae-5b99-40d4-998a-d5e2a18564e7" (UID: "e0c604ae-5b99-40d4-998a-d5e2a18564e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:30:27.498244 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.498210    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw" pod "e0c604ae-5b99-40d4-998a-d5e2a18564e7" (UID: "e0c604ae-5b99-40d4-998a-d5e2a18564e7"). InnerVolumeSpecName "kube-api-access-l58nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:30:27.596881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.596808    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:30:27.596881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.596834    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58nw\" (UniqueName: \"kubernetes.io/projected/e0c604ae-5b99-40d4-998a-d5e2a18564e7-kube-api-access-l58nw\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:30:27.596881 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:27.596844    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c604ae-5b99-40d4-998a-d5e2a18564e7-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:30:28.283948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:28.283916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc" event={"ID":"e0c604ae-5b99-40d4-998a-d5e2a18564e7","Type":"ContainerDied","Data":"2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f"}
Nov 01 01:30:28.283948 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:28.283946    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"
Nov 01 01:30:28.284193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:28.283961    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ac1491a9f892692346c273ad9f25d0aca47132f492b090e370b9ca0a24a592f"
Nov 01 01:30:31.985866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:31.985828    4493 scope.go:122] "RemoveContainer" containerID="b78ad2d07d3339686e0c2df986de6a6ffab935993c20c12e765d177b88bccd12"
Nov 01 01:30:31.994256 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:31.994233    4493 scope.go:122] "RemoveContainer" containerID="44c5c31410e77703a56a563c49789d1844644e9d08c1f8ddbfa16ccc652c1272"
Nov 01 01:30:32.002093 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:32.002075    4493 scope.go:122] "RemoveContainer" containerID="0724e4f1cbfdb31e154935664b28b1a6b7dc6098b25844ae1ecfa1ae86530a95"
Nov 01 01:30:36.781733 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.781692    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:30:36.983274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983234    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:30:36.983697 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983680    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:36.983795 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983699    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:36.983795 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983714    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e0c604ae-5b99-40d4-998a-d5e2a18564e7" containerName="revoke-keys"
Nov 01 01:30:36.983795 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983720    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c604ae-5b99-40d4-998a-d5e2a18564e7" containerName="revoke-keys"
Nov 01 01:30:36.983953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983806    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d8aba4-378b-434b-8d83-1d61892e80ae" containerName="cleanup"
Nov 01 01:30:36.983953 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.983821    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c604ae-5b99-40d4-998a-d5e2a18564e7" containerName="revoke-keys"
Nov 01 01:30:36.988635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:36.988614    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.041210 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.041100    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-1048fc-openshift-default-dockercfg-hgjs9\""
Nov 01 01:30:37.093832 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.093796    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.093839    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.093947    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094044    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094074    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094177    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"]
Nov 01 01:30:37.094296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094196    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ghp\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094339    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.094384 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.094373    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.096846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.096808    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-p5hlr"]
Nov 01 01:30:37.099815 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.099791    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:30:37.201691 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201646    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201698    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201731    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201773    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201798    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201826    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.201904 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201862    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ghp\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202305 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.201976    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202305 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202011    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202305 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202772    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202517    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202738    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.202913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.202507    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.205848 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.205820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.207396 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.207370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.264354 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.264322    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.265464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.265437    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ghp\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\") pod \"e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") " pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.304675 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.304344    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:37.753766 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.753742    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:30:37.756346 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:30:37.756319    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bbd070_c5e1_4602_bfc1_9f27a39d85a7.slice/crio-6c9768df54bb46154c526b5931c912f08347b450d71e099b33c81824442593a0 WatchSource:0}: Error finding container 6c9768df54bb46154c526b5931c912f08347b450d71e099b33c81824442593a0: Status 404 returned error can't find the container with id 6c9768df54bb46154c526b5931c912f08347b450d71e099b33c81824442593a0
Nov 01 01:30:37.758471 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.758442    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:30:37.758553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.758507    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:30:37.758553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:37.758543    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:30:38.332228 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:38.332187    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerStarted","Data":"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"}
Nov 01 01:30:38.332646 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:38.332254    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerStarted","Data":"6c9768df54bb46154c526b5931c912f08347b450d71e099b33c81824442593a0"}
Nov 01 01:30:38.453728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:38.453678    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podStartSLOduration=2.453661881 podStartE2EDuration="2.453661881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:38.452045582 +0000 UTC m=+4028.129632527" watchObservedRunningTime="2026-11-01 01:30:38.453661881 +0000 UTC m=+4028.131248826"
Nov 01 01:30:38.500869 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:38.500826    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdbfe80-27ba-4143-af54-cf3ca511ffe8" path="/var/lib/kubelet/pods/2cdbfe80-27ba-4143-af54-cf3ca511ffe8/volumes"
Nov 01 01:30:39.305844 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:39.305809    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:39.310787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:39.310762    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:39.337746 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:39.337717    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:39.338489 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:39.338473    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:30:44.990981 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:30:44.990942    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:30:55.707237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:30:55.707200    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:31:05.483436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.483404    4493 generic.go:356] "Generic (PLEG): container finished" podID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerID="bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf" exitCode=0
Nov 01 01:31:05.483892 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.483464    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerDied","Data":"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"}
Nov 01 01:31:05.483892 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.483841    4493 scope.go:122] "RemoveContainer" containerID="bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"
Nov 01 01:31:05.485056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.485025    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:31:05.485199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.485111    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:31:05.485199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:05.485186    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:31:06.491001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:06.490961    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerStarted","Data":"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"}
Nov 01 01:31:07.306205 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:07.306170    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:31:07.307445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:07.307414    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 01 01:31:07.307605 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:07.307471    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 01 01:31:08.306558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:08.306523    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 01 01:31:08.307053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:08.306591    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 01 01:31:09.306674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:09.306640    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 01 01:31:09.307095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:09.306694    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 01 01:31:09.340319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:09.340257    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:31:10.306484 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:10.306440    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 01 01:31:10.306677 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:10.306495    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 01 01:31:11.306643 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:11.306611    4493 patch_prober.go:29] interesting pod/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 01 01:31:11.307072 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:11.306664    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 01 01:31:12.310203 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:12.310172    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:31:12.311388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:12.311367    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:31:14.191020 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:14.189758    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:31:14.988965 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:31:14.988929    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:31:32.031580 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:32.031542    4493 scope.go:122] "RemoveContainer" containerID="d33627a3c220a4849ea925219cf9c3d1792ac7fa4d04b496d34b08896165ff83"
Nov 01 01:31:32.040359 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:32.040339    4493 scope.go:122] "RemoveContainer" containerID="54bc00f95a337409f3943d6fc0cf7029d968bd3a40d3b59a9305e84957ae97c6"
Nov 01 01:31:42.285325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:42.285241    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:31:44.988874 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:31:44.988840    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:31:55.278336 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.278306    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"]
Nov 01 01:31:55.281956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.281940    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.323458 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.323431    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.323625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.323464    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.323625 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.323492    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7bv\" (UniqueName: \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.334400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.334360    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:31:55.384423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.384387    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:31:55.384423 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.384397    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:31:55.384644 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.384459    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:31:55.386467 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.386446    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"]
Nov 01 01:31:55.425022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.424994    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.425022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.425023    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.425266 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.425043    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7bv\" (UniqueName: \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.425736 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.425711    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.427558 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.427536    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.486744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.486707    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7bv\" (UniqueName: \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\") pod \"maas-api-revoke-sub-06a4e38f-brx44\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.598690 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.598217    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:55.994405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:55.994379    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"]
Nov 01 01:31:55.996397 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:31:55.996371    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59eb04c_8262_4fb8_a5a2_c92a8ac5d9f1.slice/crio-c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793 WatchSource:0}: Error finding container c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793: Status 404 returned error can't find the container with id c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793
Nov 01 01:31:56.733227 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:56.733185    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44" event={"ID":"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1","Type":"ContainerStarted","Data":"e4b438a6930e469f4e34bf191a3360265e52ec06a3e6871e11c47d661d7bc807"}
Nov 01 01:31:56.733654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:56.733256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44" event={"ID":"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1","Type":"ContainerStarted","Data":"c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793"}
Nov 01 01:31:56.850792 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:56.850732    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44" podStartSLOduration=1.8507129070000001 podStartE2EDuration="1.850712907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:31:56.849496986 +0000 UTC m=+4106.527083930" watchObservedRunningTime="2026-11-01 01:31:56.850712907 +0000 UTC m=+4106.528299854"
Nov 01 01:31:57.740704 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:57.740665    4493 generic.go:356] "Generic (PLEG): container finished" podID="d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" containerID="e4b438a6930e469f4e34bf191a3360265e52ec06a3e6871e11c47d661d7bc807" exitCode=0
Nov 01 01:31:57.741114 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:57.740746    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44" event={"ID":"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1","Type":"ContainerDied","Data":"e4b438a6930e469f4e34bf191a3360265e52ec06a3e6871e11c47d661d7bc807"}
Nov 01 01:31:58.887985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.887958    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:58.957797 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.957764    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\") pod \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") "
Nov 01 01:31:58.957985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.957869    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\" (UniqueName: \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\") pod \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") "
Nov 01 01:31:58.957985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.957932    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\") pod \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\" (UID: \"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1\") "
Nov 01 01:31:58.958253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.958232    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca" pod "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" (UID: "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:31:58.960196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.960167    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca" pod "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" (UID: "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:31:58.960196 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:58.960178    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv" pod "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" (UID: "d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1"). InnerVolumeSpecName "kube-api-access-zf7bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:31:59.059019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.058940    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:31:59.059019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.058970    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:31:59.059019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.058980    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7bv\" (UniqueName: \"kubernetes.io/projected/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1-kube-api-access-zf7bv\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:31:59.751514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.751481    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44" event={"ID":"d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1","Type":"ContainerDied","Data":"c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793"}
Nov 01 01:31:59.751514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.751515    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"
Nov 01 01:31:59.751748 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:31:59.751530    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14b21ed8b6f5ce92599159f619d572b3a97431e0f05b193f71082326e09a793"
Nov 01 01:32:06.972215 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:06.972176    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"]
Nov 01 01:32:06.972754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:06.972564    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" containerName="revoke-keys"
Nov 01 01:32:06.972754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:06.972575    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" containerName="revoke-keys"
Nov 01 01:32:06.972754 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:06.972640    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" containerName="revoke-keys"
Nov 01 01:32:06.975962 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:06.975946    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.028717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.028688    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:32:07.032433 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.032406    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.032635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.032451    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.032635 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.032546    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js97x\" (UniqueName: \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.078692 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.078659    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:32:07.078908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.078772    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:32:07.078908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.078890    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:32:07.080755 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.080736    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"]
Nov 01 01:32:07.134152 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.134096    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.134354 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.134162    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.134354 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.134201    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js97x\" (UniqueName: \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.134811 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.134789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.136887 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.136863    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.195560 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.195524    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js97x\" (UniqueName: \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.289476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.289391    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:07.663765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.663740    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"]
Nov 01 01:32:07.666144 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:32:07.666103    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d68ec8e_7f07_4291_aae8_759bbedadf49.slice/crio-ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6 WatchSource:0}: Error finding container ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6: Status 404 returned error can't find the container with id ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6
Nov 01 01:32:07.790482 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:07.790446    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9" event={"ID":"9d68ec8e-7f07-4291-aae8-759bbedadf49","Type":"ContainerStarted","Data":"ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6"}
Nov 01 01:32:08.263027 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:08.262977    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:32:08.797710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:08.797619    4493 generic.go:356] "Generic (PLEG): container finished" podID="9d68ec8e-7f07-4291-aae8-759bbedadf49" containerID="8d91e21f78218f7c1fe4fd701c21a57c6eabd8e1d31fa04b57938f4f7a796305" exitCode=0
Nov 01 01:32:08.797710 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:08.797695    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9" event={"ID":"9d68ec8e-7f07-4291-aae8-759bbedadf49","Type":"ContainerDied","Data":"8d91e21f78218f7c1fe4fd701c21a57c6eabd8e1d31fa04b57938f4f7a796305"}
Nov 01 01:32:09.938933 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.938908    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:09.967849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.967815    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\") pod \"9d68ec8e-7f07-4291-aae8-759bbedadf49\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") "
Nov 01 01:32:09.968019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.967894    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\") pod \"9d68ec8e-7f07-4291-aae8-759bbedadf49\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") "
Nov 01 01:32:09.968019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.967952    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\" (UniqueName: \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\") pod \"9d68ec8e-7f07-4291-aae8-759bbedadf49\" (UID: \"9d68ec8e-7f07-4291-aae8-759bbedadf49\") "
Nov 01 01:32:09.968237 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.968214    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca" pod "9d68ec8e-7f07-4291-aae8-759bbedadf49" (UID: "9d68ec8e-7f07-4291-aae8-759bbedadf49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:32:09.968312 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.968300    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:09.969999 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.969963    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca" pod "9d68ec8e-7f07-4291-aae8-759bbedadf49" (UID: "9d68ec8e-7f07-4291-aae8-759bbedadf49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:32:09.970105 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:09.970070    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x" pod "9d68ec8e-7f07-4291-aae8-759bbedadf49" (UID: "9d68ec8e-7f07-4291-aae8-759bbedadf49"). InnerVolumeSpecName "kube-api-access-js97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:32:10.069189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:10.069086    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d68ec8e-7f07-4291-aae8-759bbedadf49-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:10.069189 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:10.069114    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js97x\" (UniqueName: \"kubernetes.io/projected/9d68ec8e-7f07-4291-aae8-759bbedadf49-kube-api-access-js97x\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:10.808669 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:10.808591    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"
Nov 01 01:32:10.808809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:10.808591    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9" event={"ID":"9d68ec8e-7f07-4291-aae8-759bbedadf49","Type":"ContainerDied","Data":"ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6"}
Nov 01 01:32:10.808809 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:10.808721    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce34884ffe921a139353185e9064e339ee2edf61f2e1bf5e7d4e1b76719fefa6"
Nov 01 01:32:14.989271 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:32:14.989237    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:32:26.721127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:26.721088    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"]
Nov 01 01:32:26.770655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:26.770623    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1048fc-8pqb9"]
Nov 01 01:32:28.501243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:28.501208    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d68ec8e-7f07-4291-aae8-759bbedadf49" path="/var/lib/kubelet/pods/9d68ec8e-7f07-4291-aae8-759bbedadf49/volumes"
Nov 01 01:32:29.045407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:29.045367    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"]
Nov 01 01:32:29.095696 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:29.095664    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f086539-4ww68"]
Nov 01 01:32:29.195731 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:29.195698    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:32:30.503240 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:30.503211    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3430fc1c-8fc8-46b6-89a4-4a20358086b5" path="/var/lib/kubelet/pods/3430fc1c-8fc8-46b6-89a4-4a20358086b5/volumes"
Nov 01 01:32:32.065569 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:32.065540    4493 scope.go:122] "RemoveContainer" containerID="e556c850166eb3830ee8b6e93b562c1c87049a39aad8fe6d4ee7519c6b715cd1"
Nov 01 01:32:32.074752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:32.074732    4493 scope.go:122] "RemoveContainer" containerID="cc06683d8f919abe66ae767f19d98f4de8df04d5f1f3f679ecc7a35b13506dbc"
Nov 01 01:32:33.359525 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:33.359496    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:32:33.359951 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:33.359723    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy" containerID="cri-o://f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0" gracePeriod=30
Nov 01 01:32:38.620614 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.620586    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:32:38.664503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664421    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664530    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664563    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664665 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664604    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664718    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:32:38.664849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664730    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664849 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664841    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.664984 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664862    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:32:38.664984 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.664879    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:32:38.665070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665047    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.665148 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665095    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.665216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665168    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\") pod \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\" (UID: \"53bbd070-c5e1-4602-bfc1-9f27a39d85a7\") "
Nov 01 01:32:38.665430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:32:38.665636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665483    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.665636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665554    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:32:38.665636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665582    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.665636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665598    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.665636 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.665613    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.667192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.667167    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "kube-api-access-x5ghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:32:38.667619 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.667598    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:32:38.667699 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.667642    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:32:38.667740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.667700    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token" pod "53bbd070-c5e1-4602-bfc1-9f27a39d85a7" (UID: "53bbd070-c5e1-4602-bfc1-9f27a39d85a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:32:38.766410 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.766377    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.766410 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.766402    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.766410 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.766412    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ghp\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-kube-api-access-x5ghp\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.766651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.766420    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.766651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.766429    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbd070-c5e1-4602-bfc1-9f27a39d85a7-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:32:38.956430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.956341    4493 generic.go:356] "Generic (PLEG): container finished" podID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerID="f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0" exitCode=0
Nov 01 01:32:38.956430 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.956417    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"
Nov 01 01:32:38.956624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.956424    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerDied","Data":"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"}
Nov 01 01:32:38.956624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.956505    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522" event={"ID":"53bbd070-c5e1-4602-bfc1-9f27a39d85a7","Type":"ContainerDied","Data":"6c9768df54bb46154c526b5931c912f08347b450d71e099b33c81824442593a0"}
Nov 01 01:32:38.956624 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.956528    4493 scope.go:122] "RemoveContainer" containerID="f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"
Nov 01 01:32:38.965719 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.965702    4493 scope.go:122] "RemoveContainer" containerID="bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"
Nov 01 01:32:38.973392 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.973372    4493 scope.go:122] "RemoveContainer" containerID="f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"
Nov 01 01:32:38.973675 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:32:38.973657    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0\": container with ID starting with f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0 not found: ID does not exist" containerID="f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"
Nov 01 01:32:38.973723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.973686    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0"} err="failed to get container status \"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0\": rpc error: code = NotFound desc = could not find container \"f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0\": container with ID starting with f7efd253cfe1656247e77d721b5601a24faa5fd12d695ae077e979b6aeeac2c0 not found: ID does not exist"
Nov 01 01:32:38.973723 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.973704    4493 scope.go:122] "RemoveContainer" containerID="bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"
Nov 01 01:32:38.973929 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:32:38.973914    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf\": container with ID starting with bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf not found: ID does not exist" containerID="bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"
Nov 01 01:32:38.973967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:38.973933    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf"} err="failed to get container status \"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf\": rpc error: code = NotFound desc = could not find container \"bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf\": container with ID starting with bb8e61ab40a6e2f84fbc1352cd5336ecda4b6efbb47ddf607d3a304884a7fdcf not found: ID does not exist"
Nov 01 01:32:39.121167 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:39.121118    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:32:39.124835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:39.124814    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1048fc-openshift-default-7b47c55b5c9p522"]
Nov 01 01:32:40.425116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425082    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:32:40.425527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425457    4493 cpu_manager.go:414] "containerMap: removing container" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:32:40.425527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425468    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:32:40.425527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425497    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9d68ec8e-7f07-4291-aae8-759bbedadf49" containerName="revoke-keys"
Nov 01 01:32:40.425527 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425502    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d68ec8e-7f07-4291-aae8-759bbedadf49" containerName="revoke-keys"
Nov 01 01:32:40.425660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425564    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:32:40.425660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.425576    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d68ec8e-7f07-4291-aae8-759bbedadf49" containerName="revoke-keys"
Nov 01 01:32:40.429744 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.429726    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481120    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481257    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481341 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481302    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481348    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481436 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481406    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsfj\" (UniqueName: \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.481520 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.481507    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.502549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.502521    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" path="/var/lib/kubelet/pods/53bbd070-c5e1-4602-bfc1-9f27a39d85a7/volumes"
Nov 01 01:32:40.532291 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.532263    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c3b10ea0-kserve-self-signed-certs\""
Nov 01 01:32:40.534273 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.534246    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.591941    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.592000    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.592049    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.592077    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsfj\" (UniqueName: \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.592112    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.592254 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.592216    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.593061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.593034    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.593397 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.593311    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.594695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.593986    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.598319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.598295    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.599471 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.599451    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.655626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.655597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsfj\" (UniqueName: \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\") pod \"e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") " pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:40.740795 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:40.740765    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:41.117338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:41.117312    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:32:41.119875 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:32:41.119846    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4a5f9d3_fd5d_43ef_81ea_11a220dd38e6.slice/crio-e39eed2b4ab4c15defd1694ce6eda070677e7e32ff67bc6bd6517950b35cc215 WatchSource:0}: Error finding container e39eed2b4ab4c15defd1694ce6eda070677e7e32ff67bc6bd6517950b35cc215: Status 404 returned error can't find the container with id e39eed2b4ab4c15defd1694ce6eda070677e7e32ff67bc6bd6517950b35cc215
Nov 01 01:32:41.972791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:41.972755    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" event={"ID":"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6","Type":"ContainerStarted","Data":"1ad9b240dee1855a1c46ed88231204c8591b721601ef61eaa51800ce2e2baf58"}
Nov 01 01:32:41.972791 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:41.972802    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" event={"ID":"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6","Type":"ContainerStarted","Data":"e39eed2b4ab4c15defd1694ce6eda070677e7e32ff67bc6bd6517950b35cc215"}
Nov 01 01:32:42.094503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:42.094450    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" podStartSLOduration=2.09443578 podStartE2EDuration="2.09443578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:32:42.09205071 +0000 UTC m=+4151.769637655" watchObservedRunningTime="2026-11-01 01:32:42.09443578 +0000 UTC m=+4151.772022762"
Nov 01 01:32:42.741683 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:42.741650    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:44.990615 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:32:44.990577    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:32:45.068688 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:45.068651    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:32:50.741405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:50.741362    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:50.753907 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:50.753884    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:32:51.039013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:32:51.038925    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:33:09.047608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:09.047576    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"]
Nov 01 01:33:09.050074 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:09.050046    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-265dj"]
Nov 01 01:33:10.501166 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:10.501113    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552130aa-f024-499e-8d4a-968f96ad6964" path="/var/lib/kubelet/pods/552130aa-f024-499e-8d4a-968f96ad6964/volumes"
Nov 01 01:33:14.989172 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:33:14.989069    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:33:27.172116 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:27.172081    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:33:32.097461 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:32.097427    4493 scope.go:122] "RemoveContainer" containerID="09ad9770c05729e61da985cf5f36110e77a65e8678feead7e343df0021780f1c"
Nov 01 01:33:44.992173 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:33:44.992114    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:33:57.465510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.465475    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"]
Nov 01 01:33:57.465891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.465832    4493 cpu_manager.go:414] "containerMap: removing container" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:33:57.465891 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.465843    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:33:57.465969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.465917    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bbd070-c5e1-4602-bfc1-9f27a39d85a7" containerName="istio-proxy"
Nov 01 01:33:57.469019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.469003    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.521716 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.521691    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:33:57.571653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.571611    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:33:57.571653 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.571641    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:33:57.571909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.571688    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:33:57.573796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.573779    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"]
Nov 01 01:33:57.630530 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.630494    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.630530 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.630535    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7pg\" (UniqueName: \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.630821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.630696    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.731400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.731363    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.731627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.731404    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7pg\" (UniqueName: \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.731627 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.731458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.732019 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.731999    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.733898 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.733876    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:57.793212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:57.793182    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7pg\" (UniqueName: \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\") pod \"maas-api-revoke-sub-452b6d7b-d2rhn\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:58.082391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:58.082314    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:33:58.455180 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:58.455156    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"]
Nov 01 01:33:58.457871 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:33:58.457842    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bf5114_9546_489a_b2bb_b34fafdb5937.slice/crio-ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b WatchSource:0}: Error finding container ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b: Status 404 returned error can't find the container with id ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b
Nov 01 01:33:58.980429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:58.980341    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:33:59.365775 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:59.365739    4493 generic.go:356] "Generic (PLEG): container finished" podID="45bf5114-9546-489a-b2bb-b34fafdb5937" containerID="f0d5bfd61a1a417f42b2d486875bb66514273915f3361b317560233c56207e3e" exitCode=0
Nov 01 01:33:59.365957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:59.365818    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn" event={"ID":"45bf5114-9546-489a-b2bb-b34fafdb5937","Type":"ContainerDied","Data":"f0d5bfd61a1a417f42b2d486875bb66514273915f3361b317560233c56207e3e"}
Nov 01 01:33:59.365957 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:33:59.365901    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn" event={"ID":"45bf5114-9546-489a-b2bb-b34fafdb5937","Type":"ContainerStarted","Data":"ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b"}
Nov 01 01:34:00.879405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:00.879359    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:34:01.011626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.010027    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\" (UniqueName: \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\") pod \"45bf5114-9546-489a-b2bb-b34fafdb5937\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") "
Nov 01 01:34:01.011626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.010111    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\") pod \"45bf5114-9546-489a-b2bb-b34fafdb5937\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") "
Nov 01 01:34:01.011626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.010235    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\") pod \"45bf5114-9546-489a-b2bb-b34fafdb5937\" (UID: \"45bf5114-9546-489a-b2bb-b34fafdb5937\") "
Nov 01 01:34:01.011626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.011515    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca" pod "45bf5114-9546-489a-b2bb-b34fafdb5937" (UID: "45bf5114-9546-489a-b2bb-b34fafdb5937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:34:01.112164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.112107    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:01.112405 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.112305    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca" pod "45bf5114-9546-489a-b2bb-b34fafdb5937" (UID: "45bf5114-9546-489a-b2bb-b34fafdb5937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:34:01.121175 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.120337    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg" pod "45bf5114-9546-489a-b2bb-b34fafdb5937" (UID: "45bf5114-9546-489a-b2bb-b34fafdb5937"). InnerVolumeSpecName "kube-api-access-kk7pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:34:01.213447 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.213413    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bf5114-9546-489a-b2bb-b34fafdb5937-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:01.213729 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.213707    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7pg\" (UniqueName: \"kubernetes.io/projected/45bf5114-9546-489a-b2bb-b34fafdb5937-kube-api-access-kk7pg\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:01.395997 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.395894    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn" event={"ID":"45bf5114-9546-489a-b2bb-b34fafdb5937","Type":"ContainerDied","Data":"ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b"}
Nov 01 01:34:01.396311 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.396274    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed887b8b687689021cd9a411bb2709350c1b4606a46b3c31194676d4e8413f2b"
Nov 01 01:34:01.396565 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:01.396550    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"
Nov 01 01:34:10.307104 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:10.307069    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:34:14.205949 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:14.205913    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:34:15.002474 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:34:15.002431    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:34:25.327188 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.327148    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" podUID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" containerName="main" containerID="cri-o://1ad9b240dee1855a1c46ed88231204c8591b721601ef61eaa51800ce2e2baf58" gracePeriod=45
Nov 01 01:34:25.539703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.539668    4493 generic.go:356] "Generic (PLEG): container finished" podID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" containerID="1ad9b240dee1855a1c46ed88231204c8591b721601ef61eaa51800ce2e2baf58" exitCode=0
Nov 01 01:34:25.539868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.539732    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" event={"ID":"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6","Type":"ContainerDied","Data":"1ad9b240dee1855a1c46ed88231204c8591b721601ef61eaa51800ce2e2baf58"}
Nov 01 01:34:25.582838 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.582785    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:34:25.709127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709099    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709164    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709190    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709335 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709268    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709361    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\" (UniqueName: \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709375    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:34:25.709514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709410    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\") pod \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\" (UID: \"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6\") "
Nov 01 01:34:25.709514 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709473    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:34:25.709698 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709570    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:34:25.709765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709733    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tmp-dir\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:25.709765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709751    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-home\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:25.709878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.709767    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-model-cache\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:25.711756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.711715    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:34:25.711756 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.711742    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "kube-api-access-sxsfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:34:25.711923 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.711839    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm" pod "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" (UID: "d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:34:25.811257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.811225    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-tls-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:25.811257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.811251    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-dshm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:25.811257 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:25.811260    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsfj\" (UniqueName: \"kubernetes.io/projected/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6-kube-api-access-sxsfj\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:26.546198 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:26.546162    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk" event={"ID":"d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6","Type":"ContainerDied","Data":"e39eed2b4ab4c15defd1694ce6eda070677e7e32ff67bc6bd6517950b35cc215"}
Nov 01 01:34:26.546584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:26.546208    4493 scope.go:122] "RemoveContainer" containerID="1ad9b240dee1855a1c46ed88231204c8591b721601ef61eaa51800ce2e2baf58"
Nov 01 01:34:26.546584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:26.546174    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"
Nov 01 01:34:26.718443 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:26.718414    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:34:26.721252 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:26.721229    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c3b10ea0-kserve-665776d96f-47ssk"]
Nov 01 01:34:28.500908 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:28.500879    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" path="/var/lib/kubelet/pods/d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6/volumes"
Nov 01 01:34:32.122272 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:32.122239    4493 scope.go:122] "RemoveContainer" containerID="89a5135be7dd0af7acee43f49bfb9dbe7edb9574ad8717fd171051bfe244b97f"
Nov 01 01:34:32.130890 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:32.130862    4493 scope.go:122] "RemoveContainer" containerID="7a291eea611804335e0eccf05dff0102d3abcd1e74c3a1ae392649b24311f780"
Nov 01 01:34:33.435247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435214    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"]
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435609    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" containerName="main"
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435621    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" containerName="main"
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435638    4493 cpu_manager.go:414] "containerMap: removing container" podUID="45bf5114-9546-489a-b2bb-b34fafdb5937" containerName="revoke-keys"
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435644    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bf5114-9546-489a-b2bb-b34fafdb5937" containerName="revoke-keys"
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435698    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a5f9d3-fd5d-43ef-81ea-11a220dd38e6" containerName="main"
Nov 01 01:34:33.435812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.435708    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bf5114-9546-489a-b2bb-b34fafdb5937" containerName="revoke-keys"
Nov 01 01:34:33.440393 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.440367    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.495501 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.495470    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:34:33.519935 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.519907    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:34:33.520159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.520061    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:34:33.520159 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.520053    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:34:33.547242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.547207    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"]
Nov 01 01:34:33.584752 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.584716    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.584922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.584783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.584922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.584830    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z2c\" (UniqueName: \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.594926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.594892    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:34:33.685812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.685732    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.685812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.685766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.685812 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.685790    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z2c\" (UniqueName: \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.686498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.686475    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.688307 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.688278    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.747077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.747043    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z2c\" (UniqueName: \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\") pod \"maas-api-revoke-sub-3e31a506-9f9dp\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:33.753918 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:33.753889    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:34.129737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:34.129535    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"]
Nov 01 01:34:34.132814 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:34:34.132770    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3302b1eb_8667_4edf_b602_7a6b58cf1452.slice/crio-c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c WatchSource:0}: Error finding container c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c: Status 404 returned error can't find the container with id c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c
Nov 01 01:34:34.600439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:34.600392    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp" event={"ID":"3302b1eb-8667-4edf-b602-7a6b58cf1452","Type":"ContainerStarted","Data":"b926da0883980cb3c0de3717f84b1bb8a809ee2553808b9aa2055822ebd519a8"}
Nov 01 01:34:34.600977 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:34.600479    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp" event={"ID":"3302b1eb-8667-4edf-b602-7a6b58cf1452","Type":"ContainerStarted","Data":"c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c"}
Nov 01 01:34:34.720654 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:34.720586    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp" podStartSLOduration=1.720565574 podStartE2EDuration="1.720565574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:34:34.719490364 +0000 UTC m=+4264.397077310" watchObservedRunningTime="2026-11-01 01:34:34.720565574 +0000 UTC m=+4264.398152518"
Nov 01 01:34:35.612402 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:35.612367    4493 generic.go:356] "Generic (PLEG): container finished" podID="3302b1eb-8667-4edf-b602-7a6b58cf1452" containerID="b926da0883980cb3c0de3717f84b1bb8a809ee2553808b9aa2055822ebd519a8" exitCode=0
Nov 01 01:34:35.613078 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:35.613051    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp" event={"ID":"3302b1eb-8667-4edf-b602-7a6b58cf1452","Type":"ContainerDied","Data":"b926da0883980cb3c0de3717f84b1bb8a809ee2553808b9aa2055822ebd519a8"}
Nov 01 01:34:36.403160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.402268    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:34:36.403160 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.402353    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:34:36.769326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.769301    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:36.862689 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.862650    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\") pod \"3302b1eb-8667-4edf-b602-7a6b58cf1452\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") "
Nov 01 01:34:36.862892 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.862724    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\") pod \"3302b1eb-8667-4edf-b602-7a6b58cf1452\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") "
Nov 01 01:34:36.862892 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.862759    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\" (UniqueName: \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\") pod \"3302b1eb-8667-4edf-b602-7a6b58cf1452\" (UID: \"3302b1eb-8667-4edf-b602-7a6b58cf1452\") "
Nov 01 01:34:36.863085 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.863058    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca" pod "3302b1eb-8667-4edf-b602-7a6b58cf1452" (UID: "3302b1eb-8667-4edf-b602-7a6b58cf1452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:34:36.864920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.864897    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca" pod "3302b1eb-8667-4edf-b602-7a6b58cf1452" (UID: "3302b1eb-8667-4edf-b602-7a6b58cf1452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:34:36.865031 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.864978    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c" pod "3302b1eb-8667-4edf-b602-7a6b58cf1452" (UID: "3302b1eb-8667-4edf-b602-7a6b58cf1452"). InnerVolumeSpecName "kube-api-access-j9z2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:34:36.963767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.963672    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:36.963767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.963719    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1eb-8667-4edf-b602-7a6b58cf1452-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:36.963767 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:36.963729    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9z2c\" (UniqueName: \"kubernetes.io/projected/3302b1eb-8667-4edf-b602-7a6b58cf1452-kube-api-access-j9z2c\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:34:37.625371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:37.625336    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp" event={"ID":"3302b1eb-8667-4edf-b602-7a6b58cf1452","Type":"ContainerDied","Data":"c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c"}
Nov 01 01:34:37.625371 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:37.625372    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"
Nov 01 01:34:37.625597 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:37.625389    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2dc63ef2598dfde343d2d046e162211073ccf07ec80008b6bb8c7b54c80f92c"
Nov 01 01:34:44.989024 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:34:44.988943    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:34:46.304944 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:46.304909    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:34:57.274721 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:57.274688    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:34:59.047343 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:59.047309    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"]
Nov 01 01:34:59.097192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:34:59.097160    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xcmhn"]
Nov 01 01:35:00.501008 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:00.500970    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204bef77-ee0c-45e3-b256-d6b1de1af7dd" path="/var/lib/kubelet/pods/204bef77-ee0c-45e3-b256-d6b1de1af7dd/volumes"
Nov 01 01:35:07.177769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.177735    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"]
Nov 01 01:35:07.178339 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.178319    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3302b1eb-8667-4edf-b602-7a6b58cf1452" containerName="revoke-keys"
Nov 01 01:35:07.178424 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.178341    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3302b1eb-8667-4edf-b602-7a6b58cf1452" containerName="revoke-keys"
Nov 01 01:35:07.178485 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.178441    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3302b1eb-8667-4edf-b602-7a6b58cf1452" containerName="revoke-keys"
Nov 01 01:35:07.182670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.182647    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.288006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.287939    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 01:35:07.288006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.287978    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:35:07.288006 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.288007    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:35:07.288303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.288024    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\""
Nov 01 01:35:07.289858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.289838    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"]
Nov 01 01:35:07.316569 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.316542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.316651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.316580    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.316651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.316611    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhjm\" (UniqueName: \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.417255 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.417226    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.417448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.417264    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.417448 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.417301    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhjm\" (UniqueName: \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.417985 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.417957    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.419843 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.419824    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.478728 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.478697    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhjm\" (UniqueName: \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\") pod \"maas-api-revoke-sub-6a15f447-vvm6g\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.495249 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.495220    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:07.866503 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:07.866424    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"]
Nov 01 01:35:07.868869 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:35:07.868836    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa19c84_c964_4ff3_92e9_fa1b2bf5d7c0.slice/crio-920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be WatchSource:0}: Error finding container 920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be: Status 404 returned error can't find the container with id 920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be
Nov 01 01:35:08.815660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:08.815626    4493 generic.go:356] "Generic (PLEG): container finished" podID="7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" containerID="1c95afd453f4f6b8a09058980f86eaed8978eeaca15ead34a0a1c66e4720f4d4" exitCode=0
Nov 01 01:35:08.816034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:08.815710    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g" event={"ID":"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0","Type":"ContainerDied","Data":"1c95afd453f4f6b8a09058980f86eaed8978eeaca15ead34a0a1c66e4720f4d4"}
Nov 01 01:35:08.816034 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:08.815766    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g" event={"ID":"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0","Type":"ContainerStarted","Data":"920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be"}
Nov 01 01:35:09.953102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:09.953079    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:10.040626 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.040592    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\") pod \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") "
Nov 01 01:35:10.040827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.040657    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\") pod \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") "
Nov 01 01:35:10.040827 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.040681    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\" (UniqueName: \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\") pod \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\" (UID: \"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0\") "
Nov 01 01:35:10.041026 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.041005    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca" pod "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" (UID: "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:35:10.042888 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.042856    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca" pod "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" (UID: "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:35:10.043001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.042940    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm" pod "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" (UID: "7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0"). InnerVolumeSpecName "kube-api-access-zrhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:35:10.142068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.141994    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-openshift-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:10.142068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.142024    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-maas-api-xks-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:10.142068 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.142035    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrhjm\" (UniqueName: \"kubernetes.io/projected/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0-kube-api-access-zrhjm\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:10.825922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.825846    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"
Nov 01 01:35:10.825922 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.825881    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g" event={"ID":"7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0","Type":"ContainerDied","Data":"920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be"}
Nov 01 01:35:10.826112 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:10.825928    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920a0695bad800adc432a3deeb03ee6eeb7570fde75807e5c4b055e3954b75be"
Nov 01 01:35:14.990597 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:35:14.990561    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:35:19.980264 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:19.980227    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:35:20.464204 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.464118    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:20.464498 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.464485    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" containerName="revoke-keys"
Nov 01 01:35:20.464540 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.464500    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" containerName="revoke-keys"
Nov 01 01:35:20.464592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.464582    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" containerName="revoke-keys"
Nov 01 01:35:20.467789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.467770    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.570389 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.570354    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 01:35:20.570389 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.570384    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jq2v\""
Nov 01 01:35:20.570651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.570406    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 01:35:20.570651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.570433    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 01:35:20.570651 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.570595    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 01:35:20.572709 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.572683    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:20.646655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.646617    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.646836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.646703    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzj7\" (UniqueName: \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.646836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.646774    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.646836 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.646806    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.747578 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.747541    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzj7\" (UniqueName: \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.747578 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.747581    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.747835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.747601    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.747835 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.747674    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.748444 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.748422    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.750217 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.750192    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.750413 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.750393    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:20.808773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:20.808746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzj7\" (UniqueName: \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\") pod \"maas-api-5c7d8966cd-nzbtd\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:21.080783 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:21.080674    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:21.460221 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:21.459756    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:21.466122 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:35:21.466092    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25478230_a415_485f_84c3_ad0f69edd452.slice/crio-dc64a061b2d80409bba53a68cc5d960b3c62fdc2d02a12cd10a2b26f07002989 WatchSource:0}: Error finding container dc64a061b2d80409bba53a68cc5d960b3c62fdc2d02a12cd10a2b26f07002989: Status 404 returned error can't find the container with id dc64a061b2d80409bba53a68cc5d960b3c62fdc2d02a12cd10a2b26f07002989
Nov 01 01:35:21.467444 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:21.467426    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 01:35:21.876613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:21.876580    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" event={"ID":"25478230-a415-485f-84c3-ad0f69edd452","Type":"ContainerStarted","Data":"dc64a061b2d80409bba53a68cc5d960b3c62fdc2d02a12cd10a2b26f07002989"}
Nov 01 01:35:22.882776 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:22.882743    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" event={"ID":"25478230-a415-485f-84c3-ad0f69edd452","Type":"ContainerStarted","Data":"73b48d2b69160171252723bd08453fd70dadeebb58275179b9d9ab28e114711f"}
Nov 01 01:35:22.883192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:22.882991    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:23.002056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:23.002002    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" podStartSLOduration=2.19868864 podStartE2EDuration="3.001986537s" totalImagesPullingTime="803.297897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:22.999770957 +0000 UTC m=+4312.677357903" watchObservedRunningTime="2026-11-01 01:35:23.001986537 +0000 UTC m=+4312.679573483"
Nov 01 01:35:27.533429 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.533395    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:35:27.536963 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.536941    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.616077 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.616031    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a684c9cb-openshift-default-dockercfg-7f6wh\""
Nov 01 01:35:27.624581 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.624846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624609    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.624846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624730    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.624846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624780    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.624846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624814    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.624846 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624841    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.625187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.624932    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjs9\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.625187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.625173    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.625325 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.625220    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.674247 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.674211    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:35:27.726192 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726120    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726224    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726265    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726591 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726377    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726547    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726656    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726696    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726729    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726830 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726759    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.726984 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.726833    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjs9\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.727227 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.727207    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.727475 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.727452    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.727971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.727947    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.729415 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.729389    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.729544 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.729525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.788062 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.787984    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.788238 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.788075    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjs9\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\") pod \"e2e-ait-a684c9cb-openshift-default-76556645c-qt656\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") " pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:27.849674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:27.849637    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:28.051388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.051272    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"]
Nov 01 01:35:28.051388 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.051346    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-c5cfc"]
Nov 01 01:35:28.286262 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.286121    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:35:28.289330 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:35:28.289297    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989139b7_bb37_45f1_a533_6fd3f0e6d149.slice/crio-a5464747891481290a9e0ecda01584544966645965264b62f923574d828df87d WatchSource:0}: Error finding container a5464747891481290a9e0ecda01584544966645965264b62f923574d828df87d: Status 404 returned error can't find the container with id a5464747891481290a9e0ecda01584544966645965264b62f923574d828df87d
Nov 01 01:35:28.291354 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.291318    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:28.291476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.291409    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:28.291476 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.291454    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:28.502304 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.502273    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c604ae-5b99-40d4-998a-d5e2a18564e7" path="/var/lib/kubelet/pods/e0c604ae-5b99-40d4-998a-d5e2a18564e7/volumes"
Nov 01 01:35:28.698749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.698648    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:28.699241 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.698878    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" podUID="25478230-a415-485f-84c3-ad0f69edd452" containerName="maas-api" containerID="cri-o://73b48d2b69160171252723bd08453fd70dadeebb58275179b9d9ab28e114711f" gracePeriod=30
Nov 01 01:35:28.710362 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.710335    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:28.732383 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.732344    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"]
Nov 01 01:35:28.735824 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.735795    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.797243 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.797206    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"]
Nov 01 01:35:28.838867 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.838826    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-maas-api-tls\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.839050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.838909    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-openshift-service-ca\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.839161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.839084    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-metrics-certs\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.839161 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.839121    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnhf\" (UniqueName: \"kubernetes.io/projected/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-kube-api-access-kvnhf\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.915230 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.915185    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerStarted","Data":"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"}
Nov 01 01:35:28.915412 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.915258    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerStarted","Data":"a5464747891481290a9e0ecda01584544966645965264b62f923574d828df87d"}
Nov 01 01:35:28.916909 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.916887    4493 generic.go:356] "Generic (PLEG): container finished" podID="25478230-a415-485f-84c3-ad0f69edd452" containerID="73b48d2b69160171252723bd08453fd70dadeebb58275179b9d9ab28e114711f" exitCode=0
Nov 01 01:35:28.917044 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.916916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" event={"ID":"25478230-a415-485f-84c3-ad0f69edd452","Type":"ContainerDied","Data":"73b48d2b69160171252723bd08453fd70dadeebb58275179b9d9ab28e114711f"}
Nov 01 01:35:28.940040 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.940013    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-metrics-certs\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.940191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.940057    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnhf\" (UniqueName: \"kubernetes.io/projected/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-kube-api-access-kvnhf\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.940191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.940112    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-maas-api-tls\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.940191 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.940172    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-openshift-service-ca\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.941095 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.941068    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-openshift-service-ca\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.943053 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.943025    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-metrics-certs\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.943174 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.943072    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-maas-api-tls\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:28.958298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:28.958232    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:29.001691 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.001655    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnhf\" (UniqueName: \"kubernetes.io/projected/c50e7b53-7f4a-418a-95c9-0dd06cffe1f2-kube-api-access-kvnhf\") pod \"maas-api-6b97d8bb58-hbgns\" (UID: \"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:29.041064 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.041028    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\") pod \"25478230-a415-485f-84c3-ad0f69edd452\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") "
Nov 01 01:35:29.041267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.041093    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\" (UniqueName: \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\") pod \"25478230-a415-485f-84c3-ad0f69edd452\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") "
Nov 01 01:35:29.041267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.041160    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\") pod \"25478230-a415-485f-84c3-ad0f69edd452\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") "
Nov 01 01:35:29.041267 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.041187    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\") pod \"25478230-a415-485f-84c3-ad0f69edd452\" (UID: \"25478230-a415-485f-84c3-ad0f69edd452\") "
Nov 01 01:35:29.041519 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.041494    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca" pod "25478230-a415-485f-84c3-ad0f69edd452" (UID: "25478230-a415-485f-84c3-ad0f69edd452"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:35:29.043421 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.043389    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs" pod "25478230-a415-485f-84c3-ad0f69edd452" (UID: "25478230-a415-485f-84c3-ad0f69edd452"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:35:29.043585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.043556    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls" pod "25478230-a415-485f-84c3-ad0f69edd452" (UID: "25478230-a415-485f-84c3-ad0f69edd452"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 01:35:29.043585 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.043569    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7" pod "25478230-a415-485f-84c3-ad0f69edd452" (UID: "25478230-a415-485f-84c3-ad0f69edd452"). InnerVolumeSpecName "kube-api-access-4hzj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:35:29.096608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.096568    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:29.142695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.142649    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzj7\" (UniqueName: \"kubernetes.io/projected/25478230-a415-485f-84c3-ad0f69edd452-kube-api-access-4hzj7\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:29.142695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.142686    4493 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-metrics-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:29.142695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.142696    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25478230-a415-485f-84c3-ad0f69edd452-maas-api-tls\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:29.142695 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.142705    4493 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25478230-a415-485f-84c3-ad0f69edd452-openshift-service-ca\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:35:29.171185 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.171105    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podStartSLOduration=2.171086308 podStartE2EDuration="2.171086308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:29.169301966 +0000 UTC m=+4318.846888911" watchObservedRunningTime="2026-11-01 01:35:29.171086308 +0000 UTC m=+4318.848673256"
Nov 01 01:35:29.473973 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.473898    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"]
Nov 01 01:35:29.476403 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:35:29.476365    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50e7b53_7f4a_418a_95c9_0dd06cffe1f2.slice/crio-0c4b70b258c1e34f058d98db617dc16fe9ef3bd1bfbf7ac57e5394756c4b3e81 WatchSource:0}: Error finding container 0c4b70b258c1e34f058d98db617dc16fe9ef3bd1bfbf7ac57e5394756c4b3e81: Status 404 returned error can't find the container with id 0c4b70b258c1e34f058d98db617dc16fe9ef3bd1bfbf7ac57e5394756c4b3e81
Nov 01 01:35:29.849969 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.849870    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:29.854916 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.854890    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:29.924164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.924106    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"
Nov 01 01:35:29.924164 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.924118    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd" event={"ID":"25478230-a415-485f-84c3-ad0f69edd452","Type":"ContainerDied","Data":"dc64a061b2d80409bba53a68cc5d960b3c62fdc2d02a12cd10a2b26f07002989"}
Nov 01 01:35:29.924420 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.924212    4493 scope.go:122] "RemoveContainer" containerID="73b48d2b69160171252723bd08453fd70dadeebb58275179b9d9ab28e114711f"
Nov 01 01:35:29.925464 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.925446    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns" event={"ID":"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2","Type":"ContainerStarted","Data":"0c4b70b258c1e34f058d98db617dc16fe9ef3bd1bfbf7ac57e5394756c4b3e81"}
Nov 01 01:35:29.925650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.925619    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:29.926596 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:29.926578    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:30.087713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:30.087685    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:30.091298 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:30.091272    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nzbtd"]
Nov 01 01:35:30.502048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:30.502017    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25478230-a415-485f-84c3-ad0f69edd452" path="/var/lib/kubelet/pods/25478230-a415-485f-84c3-ad0f69edd452/volumes"
Nov 01 01:35:30.931737 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:30.931698    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns" event={"ID":"c50e7b53-7f4a-418a-95c9-0dd06cffe1f2","Type":"ContainerStarted","Data":"e0eb93673bb47386a07f9a7cf64f2d1bde093be528cebb7a76b164a6fbcb5b47"}
Nov 01 01:35:30.932202 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:30.931806    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:31.049798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:31.049745    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns" podStartSLOduration=2.143156798 podStartE2EDuration="3.049732346s" totalImagesPullingTime="906.575548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:31.049226922 +0000 UTC m=+4320.726813868" watchObservedRunningTime="2026-11-01 01:35:31.049732346 +0000 UTC m=+4320.727319291"
Nov 01 01:35:35.100407 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.100368    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:35:35.101945 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.101905    4493 cpu_manager.go:414] "containerMap: removing container" podUID="25478230-a415-485f-84c3-ad0f69edd452" containerName="maas-api"
Nov 01 01:35:35.101945 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.101945    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25478230-a415-485f-84c3-ad0f69edd452" containerName="maas-api"
Nov 01 01:35:35.102256 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.102098    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="25478230-a415-485f-84c3-ad0f69edd452" containerName="maas-api"
Nov 01 01:35:35.107296 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.107268    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.155871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.155835    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:35:35.200598 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.200564    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.200771 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.200632    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnsxk\" (UniqueName: \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.203769 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.203751    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a684c9cb-dockercfg-smpbz\""
Nov 01 01:35:35.203864 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.203787    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a684c9cb\""
Nov 01 01:35:35.302029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.301988    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnsxk\" (UniqueName: \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.302242 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.302079    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.302764 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.302744    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.363358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.363283    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnsxk\" (UniqueName: \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\") pod \"payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") " pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.424369 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.424325    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:35.803125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.803085    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:35:35.808753 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:35:35.808721    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e7ef07_b484_4d77_b316_9c857c35e9ee.slice/crio-13934487b68c7d9e3395b2e845ebb445833dd9e3dbf7f4e8b64560fa4b3c8182 WatchSource:0}: Error finding container 13934487b68c7d9e3395b2e845ebb445833dd9e3dbf7f4e8b64560fa4b3c8182: Status 404 returned error can't find the container with id 13934487b68c7d9e3395b2e845ebb445833dd9e3dbf7f4e8b64560fa4b3c8182
Nov 01 01:35:35.958567 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.958465    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" event={"ID":"77e7ef07-b484-4d77-b316-9c857c35e9ee","Type":"ContainerStarted","Data":"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"}
Nov 01 01:35:35.958760 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.958577    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" event={"ID":"77e7ef07-b484-4d77-b316-9c857c35e9ee","Type":"ContainerStarted","Data":"13934487b68c7d9e3395b2e845ebb445833dd9e3dbf7f4e8b64560fa4b3c8182"}
Nov 01 01:35:35.958760 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:35.958607    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:36.076871 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:36.076738    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" podStartSLOduration=1.076716494 podStartE2EDuration="1.076716494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:36.074192663 +0000 UTC m=+4325.751779602" watchObservedRunningTime="2026-11-01 01:35:36.076716494 +0000 UTC m=+4325.754303438"
Nov 01 01:35:36.943278 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:36.943241    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-hbgns"
Nov 01 01:35:43.901028 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:43.900989    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:35:44.989444 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:35:44.989411    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:35:46.966150 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:46.966106    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:35:53.064125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.064047    4493 generic.go:356] "Generic (PLEG): container finished" podID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerID="2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a" exitCode=0
Nov 01 01:35:53.064572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.064118    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerDied","Data":"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"}
Nov 01 01:35:53.064572 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.064536    4493 scope.go:122] "RemoveContainer" containerID="2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"
Nov 01 01:35:53.065798 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.065769    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:53.065911 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.065879    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:53.065976 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:53.065929    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 01 01:35:54.070833 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:54.070800    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerStarted","Data":"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"}
Nov 01 01:35:54.850713 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:54.850667    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:54.851358 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:54.851330    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 01 01:35:54.851433 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:54.851397    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 01 01:35:55.850808 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:55.850769    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 01 01:35:55.851477 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:55.851445    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 01 01:35:56.851102 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:56.851063    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 01 01:35:56.851570 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:56.851169    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 01 01:35:57.850110 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:57.850068    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 01 01:35:57.850310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:57.850172    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 01 01:35:58.850742 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:58.850706    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 01 01:35:58.851295 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:58.850773    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 01 01:35:59.854447 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:59.854412    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:59.855001 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:59.854715    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:35:59.855428 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:35:59.855403    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:36:14.989200 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:36:14.989102    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:36:19.270013 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:19.269975    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:36:24.270553 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.270479    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" podUID="77e7ef07-b484-4d77-b316-9c857c35e9ee" containerName="payload-processing" containerID="cri-o://0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72" gracePeriod=25
Nov 01 01:36:24.539127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.539101    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:36:24.672303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.669235    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\" (UniqueName: \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\") pod \"77e7ef07-b484-4d77-b316-9c857c35e9ee\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") "
Nov 01 01:36:24.672303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.669336    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\") pod \"77e7ef07-b484-4d77-b316-9c857c35e9ee\" (UID: \"77e7ef07-b484-4d77-b316-9c857c35e9ee\") "
Nov 01 01:36:24.672303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.670393    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume" pod "77e7ef07-b484-4d77-b316-9c857c35e9ee" (UID: "77e7ef07-b484-4d77-b316-9c857c35e9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:36:24.672303 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.670842    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e7ef07-b484-4d77-b316-9c857c35e9ee-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:24.680662 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.680619    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk" pod "77e7ef07-b484-4d77-b316-9c857c35e9ee" (UID: "77e7ef07-b484-4d77-b316-9c857c35e9ee"). InnerVolumeSpecName "kube-api-access-xnsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:36:24.771494 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:24.771458    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnsxk\" (UniqueName: \"kubernetes.io/projected/77e7ef07-b484-4d77-b316-9c857c35e9ee-kube-api-access-xnsxk\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:25.232561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.232517    4493 generic.go:356] "Generic (PLEG): container finished" podID="77e7ef07-b484-4d77-b316-9c857c35e9ee" containerID="0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72" exitCode=0
Nov 01 01:36:25.232561 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.232576    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"
Nov 01 01:36:25.232878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.232585    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" event={"ID":"77e7ef07-b484-4d77-b316-9c857c35e9ee","Type":"ContainerDied","Data":"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"}
Nov 01 01:36:25.232878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.232637    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg" event={"ID":"77e7ef07-b484-4d77-b316-9c857c35e9ee","Type":"ContainerDied","Data":"13934487b68c7d9e3395b2e845ebb445833dd9e3dbf7f4e8b64560fa4b3c8182"}
Nov 01 01:36:25.232878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.232664    4493 scope.go:122] "RemoveContainer" containerID="0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"
Nov 01 01:36:25.254650 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.254621    4493 scope.go:122] "RemoveContainer" containerID="0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"
Nov 01 01:36:25.255057 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:36:25.255029    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72\": container with ID starting with 0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72 not found: ID does not exist" containerID="0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"
Nov 01 01:36:25.255187 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.255068    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72"} err="failed to get container status \"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72\": rpc error: code = NotFound desc = could not find container \"0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72\": container with ID starting with 0ad350e47fc2dc74c1ad4cc7b7f7a92e2429f80dfa36490bd9972c4a94d99e72 not found: ID does not exist"
Nov 01 01:36:25.397866 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.397832    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:36:25.404466 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:25.404435    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a684c9cb-55cf7445b6-2p5qg"]
Nov 01 01:36:26.501675 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:26.501642    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e7ef07-b484-4d77-b316-9c857c35e9ee" path="/var/lib/kubelet/pods/77e7ef07-b484-4d77-b316-9c857c35e9ee/volumes"
Nov 01 01:36:28.825703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:28.825672    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:36:28.826193 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:28.825881    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" containerID="cri-o://99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623" gracePeriod=30
Nov 01 01:36:28.958282 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:28.958242    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:36:29.857490 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:29.857444    4493 patch_prober.go:29] interesting pod/e2e-ait-a684c9cb-openshift-default-76556645c-qt656 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 01:36:29.857946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:29.857521    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 01:36:32.169913 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:32.169873    4493 scope.go:122] "RemoveContainer" containerID="0a8f8a3dfe2d6ad1b4fbe887cf7019391b641fd05ba9a253eca762737ad28870"
Nov 01 01:36:32.178409 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:32.178379    4493 scope.go:122] "RemoveContainer" containerID="eb43a144daa08a27a02b9b5657a0661fd1311d0d0e31c2c454e2e9da94aed4fe"
Nov 01 01:36:34.071886 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.071860    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:36:34.174584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174490    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174584 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174547    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174643    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174694    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174736    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174766    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.174839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174811    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.175125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174890    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.175125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174942    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\") pod \"989139b7-bb37-45f1-a533-6fd3f0e6d149\" (UID: \"989139b7-bb37-45f1-a533-6fd3f0e6d149\") "
Nov 01 01:36:34.175125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174967    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:36:34.175125 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.174981    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:36:34.175551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.175296    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:36:34.175551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.175388    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:36:34.175551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.175393    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-certs\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.175551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.175508    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-workload-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.175551 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.175481    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:36:34.177097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.177049    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "kube-api-access-pdjs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:36:34.177097 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.177061    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:36:34.177322 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.177117    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 01:36:34.177322 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.177161    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy" pod "989139b7-bb37-45f1-a533-6fd3f0e6d149" (UID: "989139b7-bb37-45f1-a533-6fd3f0e6d149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 01:36:34.276092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276060    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-token\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276092 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276090    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-data\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276104    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-podinfo\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276114    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjs9\" (UniqueName: \"kubernetes.io/projected/989139b7-bb37-45f1-a533-6fd3f0e6d149-kube-api-access-pdjs9\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276155    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-istio-envoy\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276166    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989139b7-bb37-45f1-a533-6fd3f0e6d149-credential-socket\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276376 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276174    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989139b7-bb37-45f1-a533-6fd3f0e6d149-istiod-ca-cert\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:36:34.276851 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276828    4493 generic.go:356] "Generic (PLEG): container finished" podID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerID="99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623" exitCode=0
Nov 01 01:36:34.276920 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276874    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"
Nov 01 01:36:34.276971 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.276875    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerDied","Data":"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"}
Nov 01 01:36:34.277041 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.277019    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656" event={"ID":"989139b7-bb37-45f1-a533-6fd3f0e6d149","Type":"ContainerDied","Data":"a5464747891481290a9e0ecda01584544966645965264b62f923574d828df87d"}
Nov 01 01:36:34.277103 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.277067    4493 scope.go:122] "RemoveContainer" containerID="99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"
Nov 01 01:36:34.288965 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.288939    4493 scope.go:122] "RemoveContainer" containerID="2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"
Nov 01 01:36:34.299679 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.299652    4493 scope.go:122] "RemoveContainer" containerID="99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"
Nov 01 01:36:34.300055 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:36:34.300030    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623\": container with ID starting with 99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623 not found: ID does not exist" containerID="99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"
Nov 01 01:36:34.300119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.300066    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623"} err="failed to get container status \"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623\": rpc error: code = NotFound desc = could not find container \"99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623\": container with ID starting with 99174311afc0dca6d5a342d9d830103c864158f69db435ae399f3c75df148623 not found: ID does not exist"
Nov 01 01:36:34.300119 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.300084    4493 scope.go:122] "RemoveContainer" containerID="2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"
Nov 01 01:36:34.300403 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:36:34.300385    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a\": container with ID starting with 2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a not found: ID does not exist" containerID="2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"
Nov 01 01:36:34.300446 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.300410    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a"} err="failed to get container status \"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a\": rpc error: code = NotFound desc = could not find container \"2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a\": container with ID starting with 2d6cec05818bf7ef4eb751e221465bec4b3b44549c3aa26f945811ad1cb1294a not found: ID does not exist"
Nov 01 01:36:34.444322 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.444238    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:36:34.448111 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.448079    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a684c9cb-openshift-default-76556645c-qt656"]
Nov 01 01:36:34.501372 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:34.501345    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" path="/var/lib/kubelet/pods/989139b7-bb37-45f1-a533-6fd3f0e6d149/volumes"
Nov 01 01:36:38.447821 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.447789    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:36:38.448319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448303    4493 cpu_manager.go:414] "containerMap: removing container" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.448370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448321    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.448370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448331    4493 cpu_manager.go:414] "containerMap: removing container" podUID="77e7ef07-b484-4d77-b316-9c857c35e9ee" containerName="payload-processing"
Nov 01 01:36:38.448370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448338    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e7ef07-b484-4d77-b316-9c857c35e9ee" containerName="payload-processing"
Nov 01 01:36:38.448370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448357    4493 cpu_manager.go:414] "containerMap: removing container" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.448370 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448362    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.448534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448421    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.448534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.448428    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e7ef07-b484-4d77-b316-9c857c35e9ee" containerName="payload-processing"
Nov 01 01:36:38.453021 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.453004    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.510460 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.510429    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:36:38.516592 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.516552    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67rq\" (UniqueName: \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.516946 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.516898    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.560445 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.560404    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-930d1a\""
Nov 01 01:36:38.560672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.560569    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-930d1a-dockercfg-5xph6\""
Nov 01 01:36:38.612391 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.612358    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:36:38.612796 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.612783    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="989139b7-bb37-45f1-a533-6fd3f0e6d149" containerName="istio-proxy"
Nov 01 01:36:38.615385 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.615360    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.618016 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.617985    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67rq\" (UniqueName: \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.618173 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.618027    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.618852 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.618825    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.663878 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.663846    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:36:38.679432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.679398    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67rq\" (UniqueName: \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\") pod \"payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.718666 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.718635    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.718841 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.718775    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7f4\" (UniqueName: \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.770899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.770868    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:38.819707 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.819675    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.819857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.819770    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7f4\" (UniqueName: \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.820411 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.820390    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.887073 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.887034    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7f4\" (UniqueName: \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\") pod \"payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:38.928183 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:38.928122    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:39.144326 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.144291    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:36:39.146010 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:36:39.145979    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd123ebd5_12e0_4dc8_bc72_61b5ac296ff2.slice/crio-c04143ebaadfc18048f866cc656149b072db8d23cc93c52167b6fecd7feb4c0f WatchSource:0}: Error finding container c04143ebaadfc18048f866cc656149b072db8d23cc93c52167b6fecd7feb4c0f: Status 404 returned error can't find the container with id c04143ebaadfc18048f866cc656149b072db8d23cc93c52167b6fecd7feb4c0f
Nov 01 01:36:39.302742 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.302711    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:36:39.303845 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.303820    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" event={"ID":"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2","Type":"ContainerStarted","Data":"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"}
Nov 01 01:36:39.303989 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.303869    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" event={"ID":"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2","Type":"ContainerStarted","Data":"c04143ebaadfc18048f866cc656149b072db8d23cc93c52167b6fecd7feb4c0f"}
Nov 01 01:36:39.303989 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.303942    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:39.304472 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:36:39.304449    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3b429e_7e4f_4f74_b760_8ffd0ca3f1c5.slice/crio-db3233e43614555b454f802fd90d52a9c0dc655a3609b68693bcc01d427ed528 WatchSource:0}: Error finding container db3233e43614555b454f802fd90d52a9c0dc655a3609b68693bcc01d427ed528: Status 404 returned error can't find the container with id db3233e43614555b454f802fd90d52a9c0dc655a3609b68693bcc01d427ed528
Nov 01 01:36:39.422674 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:39.422598    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" podStartSLOduration=1.422582048 podStartE2EDuration="1.422582048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:36:39.420485831 +0000 UTC m=+4389.098072778" watchObservedRunningTime="2026-11-01 01:36:39.422582048 +0000 UTC m=+4389.100168992"
Nov 01 01:36:40.310877 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:40.310836    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" event={"ID":"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5","Type":"ContainerStarted","Data":"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"}
Nov 01 01:36:40.311302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:40.310890    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" event={"ID":"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5","Type":"ContainerStarted","Data":"db3233e43614555b454f802fd90d52a9c0dc655a3609b68693bcc01d427ed528"}
Nov 01 01:36:40.311302 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:40.310914    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:40.429312 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:40.429235    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" podStartSLOduration=2.429215884 podStartE2EDuration="2.429215884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:36:40.427530336 +0000 UTC m=+4390.105117282" watchObservedRunningTime="2026-11-01 01:36:40.429215884 +0000 UTC m=+4390.106802832"
Nov 01 01:36:44.989505 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:36:44.989469    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:36:49.217050 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:49.217014    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:36:50.314212 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:50.314183    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:36:51.319353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:51.319325    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:36:59.058805 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:59.058765    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"]
Nov 01 01:36:59.108574 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:36:59.108547    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a4e38f-brx44"]
Nov 01 01:37:00.501179 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:37:00.501119    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1" path="/var/lib/kubelet/pods/d59eb04c-8262-4fb8-a5a2-c92a8ac5d9f1/volumes"
Nov 01 01:37:14.989254 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:37:14.989219    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:37:44.989771 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:37:44.989684    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:38:14.989743 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:38:14.989704    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:38:32.215590 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:32.215563    4493 scope.go:122] "RemoveContainer" containerID="8d91e21f78218f7c1fe4fd701c21a57c6eabd8e1d31fa04b57938f4f7a796305"
Nov 01 01:38:32.223613 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:32.223592    4493 scope.go:122] "RemoveContainer" containerID="e4b438a6930e469f4e34bf191a3360265e52ec06a3e6871e11c47d661d7bc807"
Nov 01 01:38:32.777926 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:32.777884    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:38:32.884071 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:32.883984    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:38:37.778857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:37.778788    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" podUID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" containerName="payload-processing" containerID="cri-o://4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015" gracePeriod=25
Nov 01 01:38:37.885400 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:37.885355    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" podUID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" containerName="payload-pre-processing" containerID="cri-o://3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe" gracePeriod=25
Nov 01 01:38:38.043685 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.043616    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:38:38.088672 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.088637    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\") pod \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") "
Nov 01 01:38:38.089108 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.089082    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume" pod "7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" (UID: "7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:38:38.156933 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.156908    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:38:38.189820 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.189787    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\") pod \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") "
Nov 01 01:38:38.190030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.189837    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\" (UniqueName: \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\") pod \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\" (UID: \"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2\") "
Nov 01 01:38:38.190030 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.189877    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\" (UniqueName: \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\") pod \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\" (UID: \"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5\") "
Nov 01 01:38:38.190199 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.190092    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:38:38.190280 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.190242    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume" pod "d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" (UID: "d123ebd5-12e0-4dc8-bc72-61b5ac296ff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 01:38:38.192329 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.192299    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq" pod "d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" (UID: "d123ebd5-12e0-4dc8-bc72-61b5ac296ff2"). InnerVolumeSpecName "kube-api-access-n67rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:38:38.192439 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.192322    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4" pod "7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" (UID: "7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5"). InnerVolumeSpecName "kube-api-access-9h7f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 01:38:38.291263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.291231    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-plugins-config-volume\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:38:38.291263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.291261    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67rq\" (UniqueName: \"kubernetes.io/projected/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2-kube-api-access-n67rq\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:38:38.291263 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.291270    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7f4\" (UniqueName: \"kubernetes.io/projected/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5-kube-api-access-9h7f4\") on node \"ip-10-0-141-75.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 01:38:38.883338 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.883302    4493 generic.go:356] "Generic (PLEG): container finished" podID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" containerID="4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015" exitCode=0
Nov 01 01:38:38.883826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.883359    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"
Nov 01 01:38:38.883826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.883362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" event={"ID":"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5","Type":"ContainerDied","Data":"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"}
Nov 01 01:38:38.883826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.883425    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t" event={"ID":"7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5","Type":"ContainerDied","Data":"db3233e43614555b454f802fd90d52a9c0dc655a3609b68693bcc01d427ed528"}
Nov 01 01:38:38.883826 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.883448    4493 scope.go:122] "RemoveContainer" containerID="4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"
Nov 01 01:38:38.885002 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.884981    4493 generic.go:356] "Generic (PLEG): container finished" podID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" containerID="3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe" exitCode=0
Nov 01 01:38:38.885101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.885030    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"
Nov 01 01:38:38.885101 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.885039    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" event={"ID":"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2","Type":"ContainerDied","Data":"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"}
Nov 01 01:38:38.885284 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.885266    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w" event={"ID":"d123ebd5-12e0-4dc8-bc72-61b5ac296ff2","Type":"ContainerDied","Data":"c04143ebaadfc18048f866cc656149b072db8d23cc93c52167b6fecd7feb4c0f"}
Nov 01 01:38:38.892310 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.892294    4493 scope.go:122] "RemoveContainer" containerID="4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"
Nov 01 01:38:38.892624 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:38:38.892601    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015\": container with ID starting with 4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015 not found: ID does not exist" containerID="4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"
Nov 01 01:38:38.892670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.892634    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015"} err="failed to get container status \"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015\": rpc error: code = NotFound desc = could not find container \"4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015\": container with ID starting with 4f63aa42b4d399aacd47b579b8163173bdbf86594063ef46d8830c7acacd4015 not found: ID does not exist"
Nov 01 01:38:38.892670 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.892650    4493 scope.go:122] "RemoveContainer" containerID="3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"
Nov 01 01:38:38.900556 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.900537    4493 scope.go:122] "RemoveContainer" containerID="3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"
Nov 01 01:38:38.900810 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:38:38.900789    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe\": container with ID starting with 3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe not found: ID does not exist" containerID="3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"
Nov 01 01:38:38.900858 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:38.900823    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe"} err="failed to get container status \"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe\": rpc error: code = NotFound desc = could not find container \"3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe\": container with ID starting with 3cd103e29af3f7781398d5a8a742d870c6728d546a77bcdffedc490a497f37fe not found: ID does not exist"
Nov 01 01:38:39.048230 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:39.048147    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:38:39.098319 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:39.098283    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-930d1a-69b6768bf9-xcj5w"]
Nov 01 01:38:39.163689 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:39.163656    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:38:39.168740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:39.168709    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-930d1a-74799b4944-jzq8t"]
Nov 01 01:38:40.501741 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:40.501707    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" path="/var/lib/kubelet/pods/7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5/volumes"
Nov 01 01:38:40.502151 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:40.502070    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" path="/var/lib/kubelet/pods/d123ebd5-12e0-4dc8-bc72-61b5ac296ff2/volumes"
Nov 01 01:38:44.989622 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:38:44.989580    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:38:48.464773 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:48.464738    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:38:59.487048 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:38:59.487013    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:39:02.068165 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:02.067322    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"]
Nov 01 01:39:02.116216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:02.116182    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2rhn"]
Nov 01 01:39:02.510294 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:02.510252    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bf5114-9546-489a-b2bb-b34fafdb5937" path="/var/lib/kubelet/pods/45bf5114-9546-489a-b2bb-b34fafdb5937/volumes"
Nov 01 01:39:11.512353 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:11.512316    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:39:14.989172 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:39:14.989105    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:39:32.578550 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:32.578466    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:39:37.051765 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:37.051730    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"]
Nov 01 01:39:37.104610 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:37.104570    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9f9dp"]
Nov 01 01:39:38.504831 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:39:38.504797    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3302b1eb-8667-4edf-b602-7a6b58cf1452" path="/var/lib/kubelet/pods/3302b1eb-8667-4edf-b602-7a6b58cf1452/volumes"
Nov 01 01:39:44.989481 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:39:44.989438    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:40:04.401178 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:04.401116    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8587cb6b5c-nb977_808deacd-de73-48da-8649-28c36316e280/authorino/0.log"
Nov 01 01:40:04.899834 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:04.899798    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v4cj6"]
Nov 01 01:40:06.389290 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:06.389242    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6vs77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 01:40:06.389938 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:06.389902    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6vs77" podUID="3811c0a7-1656-40ee-b264-2bb5a065ed92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 01:40:11.047231 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:11.047191    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"]
Nov 01 01:40:11.096998 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:11.096966    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a15f447-vvm6g"]
Nov 01 01:40:12.501061 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:12.501024    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0" path="/var/lib/kubelet/pods/7aa19c84-c964-4ff3-92e9-fa1b2bf5d7c0/volumes"
Nov 01 01:40:14.989090 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:40:14.989054    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:40:17.577418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:17.577385    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qxkrv_fdb7b4c5-a144-4029-a2f2-46895f82ef7a/manager/0.log"
Nov 01 01:40:18.802608 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:18.802577    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b97d8bb58-hbgns_c50e7b53-7f4a-418a-95c9-0dd06cffe1f2/maas-api/0.log"
Nov 01 01:40:21.742418 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:21.742383    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8587cb6b5c-nb977_808deacd-de73-48da-8649-28c36316e280/authorino/0.log"
Nov 01 01:40:23.497153 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:23.497103    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v4cj6_8cc39922-b719-4e48-8907-ee00f259cb9c/limitador/0.log"
Nov 01 01:40:24.838708 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:24.838670    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x62wb_044a721d-67d4-4b5d-98ee-49efb2d0d746/discovery/0.log"
Nov 01 01:40:25.402967 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:25.402937    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c59cdf59-jk54h_29781e95-2467-4abd-ba34-e73bf9812f2f/kube-auth-proxy/0.log"
Nov 01 01:40:25.640052 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:25.640023    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-48lrj_bc06b95d-b4c0-4dd9-822e-b0df3090ee29/istio-proxy/0.log"
Nov 01 01:40:26.156023 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:26.155993    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-6phck_22faf631-2efa-45a5-b54b-db37f6ef4c3a/istio-proxy/0.log"
Nov 01 01:40:26.427534 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:26.427459    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-6vs77_3811c0a7-1656-40ee-b264-2bb5a065ed92/istio-proxy/0.log"
Nov 01 01:40:27.647706 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:27.647672    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-qt8tv_d0db8044-5359-45ce-8f45-6fc263a69edb/istio-proxy/0.log"
Nov 01 01:40:28.422274 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:28.422242    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-cr9md_2795bb4c-0811-45a4-b027-d436e65d481b/payload-pre-processing/0.log"
Nov 01 01:40:28.642526 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:28.642501    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:40:30.700499 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:30.700471    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6wztl_18a2f4eb-14ce-4b2a-9595-1df6fcf4576e/main/0.log"
Nov 01 01:40:30.984717 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:30.984683    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-q25ns_70817335-7e72-4250-b50d-acb0c8adbda5/main/0.log"
Nov 01 01:40:31.217778 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:31.217743    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r_7b66841e-e506-4f6f-bc1a-c79da07fbde6/main/0.log"
Nov 01 01:40:31.682512 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:31.682481    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k_775aeac2-6bb0-4e45-9019-2e09e6e8b4f5/main/0.log"
Nov 01 01:40:32.262107 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:32.262075    4493 scope.go:122] "RemoveContainer" containerID="f0d5bfd61a1a417f42b2d486875bb66514273915f3361b317560233c56207e3e"
Nov 01 01:40:33.004349 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:33.004319    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8587cb6b5c-nb977_808deacd-de73-48da-8649-28c36316e280/authorino/0.log"
Nov 01 01:40:44.533301 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:44.533224    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qxkrv_fdb7b4c5-a144-4029-a2f2-46895f82ef7a/manager/0.log"
Nov 01 01:40:44.990429 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:40:44.990398    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:40:45.427510 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:45.427431    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b97d8bb58-hbgns_c50e7b53-7f4a-418a-95c9-0dd06cffe1f2/maas-api/0.log"
Nov 01 01:40:47.785181 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:47.785147    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8587cb6b5c-nb977_808deacd-de73-48da-8649-28c36316e280/authorino/0.log"
Nov 01 01:40:49.462789 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:49.462696    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v4cj6_8cc39922-b719-4e48-8907-ee00f259cb9c/limitador/0.log"
Nov 01 01:40:50.543925 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:50.543889    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x62wb_044a721d-67d4-4b5d-98ee-49efb2d0d746/discovery/0.log"
Nov 01 01:40:51.146964 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:51.146938    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c59cdf59-jk54h_29781e95-2467-4abd-ba34-e73bf9812f2f/kube-auth-proxy/0.log"
Nov 01 01:40:51.372857 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:51.372826    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-48lrj_bc06b95d-b4c0-4dd9-822e-b0df3090ee29/istio-proxy/0.log"
Nov 01 01:40:51.874899 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:51.874865    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-6phck_22faf631-2efa-45a5-b54b-db37f6ef4c3a/istio-proxy/0.log"
Nov 01 01:40:52.096127 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:52.096093    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-6vs77_3811c0a7-1656-40ee-b264-2bb5a065ed92/istio-proxy/0.log"
Nov 01 01:40:53.323703 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:53.323677    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-qt8tv_d0db8044-5359-45ce-8f45-6fc263a69edb/istio-proxy/0.log"
Nov 01 01:40:54.091740 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:54.091707    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-cr9md_2795bb4c-0811-45a4-b027-d436e65d481b/payload-pre-processing/0.log"
Nov 01 01:40:54.344868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:54.344789    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:40:55.779056 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:55.779028    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6wztl_18a2f4eb-14ce-4b2a-9595-1df6fcf4576e/main/0.log"
Nov 01 01:40:56.054360 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:56.054282    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-q25ns_70817335-7e72-4250-b50d-acb0c8adbda5/main/0.log"
Nov 01 01:40:56.269171 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:56.269121    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtl7r_7b66841e-e506-4f6f-bc1a-c79da07fbde6/main/0.log"
Nov 01 01:40:56.735641 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:40:56.735607    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fzp4k_775aeac2-6bb0-4e45-9019-2e09e6e8b4f5/main/0.log"
Nov 01 01:41:14.987983 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:41:14.987949    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:41:26.742549 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:41:26.742520    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:41:26.982523 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:41:26.982495    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:41:32.281771 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:41:32.281734    4493 scope.go:122] "RemoveContainer" containerID="1c95afd453f4f6b8a09058980f86eaed8978eeaca15ead34a0a1c66e4720f4d4"
Nov 01 01:41:32.290562 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:41:32.290545    4493 scope.go:122] "RemoveContainer" containerID="b926da0883980cb3c0de3717f84b1bb8a809ee2553808b9aa2055822ebd519a8"
Nov 01 01:41:44.989785 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:41:44.989745    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:42:05.671153 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:05.671112    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:42:05.886299 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:05.886267    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-jdwxc_2f2595c2-9fd3-4764-a501-e137bbabf621/payload-processing/0.log"
Nov 01 01:42:14.989425 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:42:14.989340    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:42:15.944606 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:15.944575    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q456m_3554c745-2c9d-43a8-a83e-50089377ab99/global-pull-secret-syncer/0.log"
Nov 01 01:42:19.083749 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:19.083704    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hcfh5_2affb2ab-ec09-43ea-8699-8654990567f3/konnectivity-agent/0.log"
Nov 01 01:42:20.318416 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:20.318378    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-75.us-west-1.compute.internal_e15a76296c7098967185b7bb6ae1aa66/haproxy/0.log"
Nov 01 01:42:24.811022 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:24.810984    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8587cb6b5c-nb977_808deacd-de73-48da-8649-28c36316e280/authorino/0.log"
Nov 01 01:42:25.401432 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401398    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"]
Nov 01 01:42:25.401832 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401816    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" containerName="payload-pre-processing"
Nov 01 01:42:25.401882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401834    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" containerName="payload-pre-processing"
Nov 01 01:42:25.401882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401849    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" containerName="payload-processing"
Nov 01 01:42:25.401882 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401856    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" containerName="payload-processing"
Nov 01 01:42:25.401979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401921    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d123ebd5-12e0-4dc8-bc72-61b5ac296ff2" containerName="payload-pre-processing"
Nov 01 01:42:25.401979 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.401931    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3b429e-7e4f-4f74-b760-8ffd0ca3f1c5" containerName="payload-processing"
Nov 01 01:42:25.404956 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.404939    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.457521 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.457476    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"openshift-service-ca.crt\""
Nov 01 01:42:25.500687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.500656    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-lib-modules\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.500687 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.500692    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-podres\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.500916 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.500728    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-kube-api-access-grthv\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.500916 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.500798    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-proc\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.501029 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.500919    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-sys\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.507496 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.507463    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"kube-root-ca.crt\""
Nov 01 01:42:25.507660 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.507524    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd5lg\"/\"default-dockercfg-6z6n4\""
Nov 01 01:42:25.509216 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.509193    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"]
Nov 01 01:42:25.601868 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.601836    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-lib-modules\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.601876    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-podres\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.601963    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-podres\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.601984    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-lib-modules\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.602024    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-kube-api-access-grthv\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602070 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.602062    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-proc\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.602207    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-proc\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.602244    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-sys\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.602395 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.602325    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-sys\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.663595 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.663519    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/aa1ec42f-9faf-4310-8c8a-ed994f1f088d-kube-api-access-grthv\") pod \"perf-node-gather-daemonset-95t57\" (UID: \"aa1ec42f-9faf-4310-8c8a-ed994f1f088d\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:25.717705 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:25.717669    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:26.093850 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:26.093827    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"]
Nov 01 01:42:26.097024 ip-10-0-141-75 kubenswrapper[4493]: W1101 01:42:26.096994    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa1ec42f_9faf_4310_8c8a_ed994f1f088d.slice/crio-17ada40ef0baa98af8e2c7d315f948546d9e03d0fdae1131c50c43254132aa4a WatchSource:0}: Error finding container 17ada40ef0baa98af8e2c7d315f948546d9e03d0fdae1131c50c43254132aa4a: Status 404 returned error can't find the container with id 17ada40ef0baa98af8e2c7d315f948546d9e03d0fdae1131c50c43254132aa4a
Nov 01 01:42:26.920622 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:26.920591    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v4cj6_8cc39922-b719-4e48-8907-ee00f259cb9c/limitador/0.log"
Nov 01 01:42:26.956493 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:26.956460    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57" event={"ID":"aa1ec42f-9faf-4310-8c8a-ed994f1f088d","Type":"ContainerStarted","Data":"cecdaf1cf8200d18298fd61a5f47be683a23913482c031657eca18fb20d19f47"}
Nov 01 01:42:26.956694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:26.956516    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57" event={"ID":"aa1ec42f-9faf-4310-8c8a-ed994f1f088d","Type":"ContainerStarted","Data":"17ada40ef0baa98af8e2c7d315f948546d9e03d0fdae1131c50c43254132aa4a"}
Nov 01 01:42:26.956694 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:26.956577    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:27.073253 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:27.073203    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57" podStartSLOduration=2.073189671 podStartE2EDuration="2.073189671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 01:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:42:27.071284388 +0000 UTC m=+4736.748871334" watchObservedRunningTime="2026-11-01 01:42:27.073189671 +0000 UTC m=+4736.750776616"
Nov 01 01:42:32.971655 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:32.971625    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-95t57"
Nov 01 01:42:38.059787 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:38.059756    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpmn4_2aef2bdc-97cd-4716-887e-b7756b67aeca/node-exporter/0.log"
Nov 01 01:42:38.478060 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:38.478033    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpmn4_2aef2bdc-97cd-4716-887e-b7756b67aeca/kube-rbac-proxy/0.log"
Nov 01 01:42:38.896524 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:38.896449    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpmn4_2aef2bdc-97cd-4716-887e-b7756b67aeca/init-textfile/0.log"
Nov 01 01:42:44.380847 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:44.380814    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-phszk_3f15387e-7a26-4540-ab83-9d7dc0e9a650/prometheus-operator-admission-webhook/0.log"
Nov 01 01:42:44.803839 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:44.803802    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4d9d98cb-czmcl_61859786-7308-482c-8959-38741e6a97ad/telemeter-client/0.log"
Nov 01 01:42:44.989290 ip-10-0-141-75 kubenswrapper[4493]: E1101 01:42:44.989257    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd52b" containerName="extractor"
Nov 01 01:42:45.228715 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:45.228687    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4d9d98cb-czmcl_61859786-7308-482c-8959-38741e6a97ad/reload/0.log"
Nov 01 01:42:45.654508 ip-10-0-141-75 kubenswrapper[4493]: I1101 01:42:45.654420    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4d9d98cb-czmcl_61859786-7308-482c-8959-38741e6a97ad/kube-rbac-proxy/0.log"